Systems, methods, and apparatuses for populating a table having null values using a predictive query interface

ABSTRACT

Disclosed herein are systems and methods for populating a table having null values using a predictive query interface including means for receiving a tabular dataset from a user as input, the tabular dataset having data values organized as columns and rows; identifying a plurality of null values within the tabular dataset, the null values being dispersed across multiple rows and multiple columns of the tabular dataset; generating indices from the tabular dataset of columns and rows, the indices representing probabilistic relationships between the rows and the columns of the tabular dataset; displaying the tabular dataset as output to the user, the displayed output including the data values depicted as known values and the null values depicted as unknown values; receiving input from the user to populate at least a portion of the unknown values within the displayed tabular dataset with predicted values; querying the indices for the predicted values; and displaying the predicted values as updated output to the user. Other related embodiments are further disclosed

CLAIM OF PRIORITY

This United States non-provisional utility patent application is relatedto, and claims priority to, the provisional application entitled“SYSTEMS AND METHODS FOR PREDICTIVE QUERY IMPLEMENTATION AND USAGE IN AMULTI-TENANT DATABASE SYSTEM,” filed on Mar. 13, 2013, having anapplication number of 61/780,503 and attorney docket No. 8956P119Z, theentire contents of which are incorporated herein by reference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

TECHNICAL FIELD

Embodiments relate generally to the field of computing, and moreparticularly, to systems and methods for populating a table having nullvalues using a predictive query interface.

BACKGROUND

The subject matter discussed in the background section should not beassumed to be prior art merely as a result of its mention in thebackground section. Similarly, a problem mentioned in the backgroundsection or associated with the subject matter of the background sectionshould not be assumed to have been previously recognized in the priorart. The subject matter in the background section merely representsdifferent approaches, which in and of themselves may also correspond toclaimed embodiments.

Client organizations with datasets in their databases may benefit frompredictive analysis. Unfortunately, there is no low cost and scalablesolution in the marketplace today. Instead, client organizations musthire technical experts to develop customized mathematical constructs andpredictive models which are very expensive. Consequently, clientorganizations without vast financial means are simply priced out of themarket and thus do not have access to predictive analysis capabilitiesfor their datasets.

Client organizations that have the financial means to hire technical andmathematical experts to develop the necessary mathematical constructsand predictive models suffer from a common problem with customizedsolutions. Specifically, the customized solution is tailored to theparticular problem at hand at a given point in time, and as such, thecustomized solution is not able to accommodate changes to the underlyingdata structure, the customized solution is not able to accommodatechanges to the types of data stored within the client's datasets, nor isthe customized solution able to scale up to meet increasing and changingdemands of the client as their business and dataset grows over time.

The present state of the art may therefore benefit from systems andmethods for predictive query implementation and usage in an on-demandand/or multi-tenant database system as described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example, and not by way oflimitation, and will be more fully understood with reference to thefollowing detailed description when considered in connection with thefigures in which:

FIG. 1 depicts an exemplary architecture in accordance with describedembodiments;

FIG. 2 illustrates a block diagram of an example of an environment inwhich an on-demand database service might be used;

FIG. 3 illustrates a block diagram of an embodiment of elements of FIG.2 and various possible interconnections between these elements;

FIG. 4 illustrates a diagrammatic representation of a machine in theexemplary form of a computer system, in accordance with one embodiment;

FIG. 5A depicts a tablet computing device and a hand-held smartphoneeach having a circuitry integrated therein as described in accordancewith the embodiments;

FIG. 5B is a block diagram of an embodiment of tablet computing device,a smart phone, or other mobile device in which touchscreen interfaceconnectors are used;

FIG. 6 depicts a simplified flow for probabilistic modeling;

FIG. 7 illustrates an exemplary landscape upon which a random walk maybe performed;

FIG. 8 depicts an exemplary tabular dataset;

FIG. 9 depicts means for deriving motivation or causal relationshipsbetween observed data;

FIG. 10A depicts an exemplary cross-categorization in still furtherdetail;

FIG. 10B depicts an assessment of convergence, showing inferred versusground truth;

FIG. 11 depicts a chart and graph of the Bell number series;

FIG. 12A depicts an exemplary cross categorization of a small tabulardataset;

FIG. 12B depicts an exemplary architecture having implemented dataupload, processing, and predictive query API exposure in accordance withdescribed embodiments;

FIG. 12C is a flow diagram illustrating a method for implementing dataupload, processing, and predictive query API exposure in accordance withdisclosed embodiments;

FIG. 12D depicts an exemplary architecture having implemented predictivequery interface as a cloud service in accordance with describedembodiments;

FIG. 12E is a flow diagram illustrating a method for implementingpredictive query interface as a cloud service in accordance withdisclosed embodiments;

FIG. 13A illustrates usage of the RELATED command term in accordancewith the described embodiments;

FIG. 13B depicts an exemplary architecture in accordance with describedembodiments;

FIG. 13C is a flow diagram illustrating a method in accordance withdisclosed embodiments;

FIG. 14A illustrates usage of the GROUP command term in accordance withthe described embodiments;

FIG. 14B depicts an exemplary architecture in accordance with describedembodiments;

FIG. 14C is a flow diagram illustrating a method in accordance withdisclosed embodiments;

FIG. 15A illustrates usage of the SIMILAR command term in accordancewith the described embodiments;

FIG. 15B depicts an exemplary architecture in accordance with describedembodiments;

FIG. 15C is a flow diagram illustrating a method in accordance withdisclosed embodiments;

FIG. 16A illustrates usage of the PREDICT command term in accordancewith the described embodiments;

FIG. 16B illustrates usage of the PREDICT command term in accordancewith the described embodiments;

FIG. 16C illustrates usage of the PREDICT command term in accordancewith the described embodiments;

FIG. 16D depicts an exemplary architecture in accordance with describedembodiments;

FIG. 16E is a flow diagram illustrating a method in accordance withdisclosed embodiments;

FIG. 16F depicts an exemplary architecture in accordance with describedembodiments;

FIG. 16G is a flow diagram illustrating a method in accordance withdisclosed embodiments;

FIG. 17A depicts a Graphical User Interface (GUI) to display andmanipulate a tabular dataset having missing values by exploiting aPREDICT command term;

FIG. 17B depicts another view of the Graphical User Interface;

FIG. 17C depicts another view of the Graphical User Interface;

FIG. 17D depicts an exemplary architecture in accordance with describedembodiments;

FIG. 17E is a flow diagram illustrating a method in accordance withdisclosed embodiments;

FIG. 18 depicts feature moves and entity moves within indices generatedfrom analysis of tabular datasets;

FIG. 19A depicts a specialized GUI to query using historical dates;

FIG. 19B depicts an additional view of a specialized GUI to query usinghistorical dates;

FIG. 19C depicts another view of a specialized GUI to configurepredictive queries;

FIG. 19D depicts an exemplary architecture in accordance with describedembodiments;

FIG. 19E is a flow diagram illustrating a method in accordance withdisclosed embodiments;

FIG. 20A depicts a pipeline change report in accordance with describedembodiments;

FIG. 20B depicts a waterfall chart using predictive data in accordancewith described embodiments;

FIG. 20C depicts an interface with defaults after adding a firsthistorical field;

FIG. 20D depicts in additional detail an interface with defaults for anadded custom filter;

FIG. 20E depicts another interface with defaults for an added customfilter;

FIG. 20F depicts an exemplary architecture in accordance with describedembodiments;

FIG. 20G is a flow diagram illustrating a method in accordance withdisclosed embodiments;

FIG. 21A provides a chart depicting prediction completeness versusaccuracy;

FIG. 21B provides a chart depicting an opportunity confidence breakdown;

FIG. 21C provides a chart depicting an opportunity win prediction;

FIG. 22A provides a chart depicting predictive relationships foropportunity scoring;

FIG. 22B provides another chart depicting predictive relationships foropportunity scoring; and

FIG. 22C provides another chart depicting predictive relationships foropportunity scoring.

DETAILED DESCRIPTION

Client organizations who desire to perform predictive analytics and datamining against their datasets must normally hire technical experts andexplain the problem they wish to solve and then turn their data over tothe hired experts to apply customized mathematical constructs in anattempt to solve the problem at hand.

By analogy, many years ago when computer engineers designed a computersystem it was necessary to also figure out how to map data onto aphysical disk, accounting for sectors, blocks, rotational speed, etc.Modern programmers simply do not concern themselves with such issues.Similarly, it is highly desirable to utilize a server and sophisticateddatabase technology to perform data analytics for ordinary users withouthaving to hire specialized experts. By doing so, resources may be freedup to focus on other problems. The methodologies described hereinadvance the art of predictive queries toward that goal by providingsystems and methods for predictive query implementation and usage in anon-demand and/or multi-tenant database system. These methodologies movemuch of the mathematical and technological complexity into a hosteddatabase system and thus out of the view of the users. In doing so, thelearning curve to novice users is reduced and thus, the predictivetechnology is made available to a greater swath of the market place.

Certain machine learning capabilities exist today. For instance, presentcapabilities may predictively answer questions such as, “Is this persongoing to buy product x?” But existing technologies are not practicalwhen addressing a wide range of problems. For instance, a largehealthcare corporation with vast financial resources may be able to hiretechnical experts to develop customized analytics to solve a specificproblem based on the large healthcare corporations' local proprietarydatabase, but a small company by contrast simply cannot afford to hiresuch service providers as the cost far outweighs a small company'sfinancial resources to do so. Moreover, as alluded to above, even if anorganization invests in such a customized solution, that solution isforever locked to the specific problem solved and cannot scale to newproblems, new inquiries, changing data types or data structures, and soforth. As such, the custom developed solution will decay over time as itbecomes less aligned to the new and ever changing business objectives ofthe organization. Consequently, the exemplary small company must foregosolving the problem at hand whereas the entity having hired experts todevelop a custom solution are forced to re-invest additional time andresources to update and re-tool their customized solution as businessconditions, data, and objectives change over time. Neither outcome isideal.

The services offered by technical experts in the field of analytics andpredictive modeling today provide solutions that are customized to theparticular dataset of the customer. They do not offer capabilities thatmay be used by non-experts nor do they offer solutions that areabstracted from a particular underlying dataset. Instead, the modelsdeveloped require specialized training not just to implement, but toutilize, and such models are anchored to the particular underlyingdataset for which they are developed.

Conversely, the methodologies described herein provide a foundationalarchitecture by which the variously described query techniques,interfaces, databases, and other functionality is suitable for use by awide array of customer organizations and users of varying level ofexpertise as well as underlying datasets of varying scope.

Salesforce.com provides on-demand cloud services to clients,organizations, and end users, and behind those cloud services is amulti-tenant database system which permits users to have customizeddata, customized field types, and so forth. The underlying data and datastructures are customized by the client organizations for their ownparticular needs. The methodologies described herein are neverthelesscapable of analyzing and querying those datasets and data structuresbecause the methodologies are not anchored to any particular underlyingdatabase scheme, structure, or content.

Customer organizations using the described techniques further benefitfrom the low cost of access made possible by the high scalability of thesolutions described. For instance, the cloud service provider may electto provide the capability as part of an overall service offering at noadditional cost, or may elect to provide the additional capabilities foran additional service fee. In either case, customer organizations arenot required to invest a large sum up front for a one-time customizedsolution as is the case with conventional techniques. Because thecapabilities may be systematically integrated into a cloud service'scomputing architecture and because they do not require experts to customtailor solutions for each particular client organizations' dataset andstructure, the scalability brings massive cost savings, thus enablingeven small organizations with limited financial resources to benefitfrom predictive query and latent structure query techniques. Largecompanies with the financial means may also benefit due to the costsavings available to them and may further benefit from the capability toinstitute predictive query and latent structure query techniques for amuch larger array of inquiry than was previously feasible utilizingconventional techniques.

Theses and other benefits as well as more specific embodiments aredescribed in greater detail below. In the following description,numerous specific details are set forth such as examples of specificsystems, languages, components, etc., in order to provide a thoroughunderstanding of the various embodiments. It will be apparent, however,to one skilled in the art that these specific details need not beemployed to practice the embodiments disclosed herein. In otherinstances, well known materials or methods have not been described indetail in order to avoid unnecessarily obscuring the disclosedembodiments.

In addition to various hardware components depicted in the figures anddescribed herein, embodiments further include various operations whichare described below. The operations described in accordance with suchembodiments may be performed by hardware components or may be embodiedin machine-executable instructions, which may be used to cause ageneral-purpose or special-purpose processor programmed with theinstructions to perform the operations. Alternatively, the operationsmay be performed by a combination of hardware and software.

Embodiments also relate to an apparatus for performing the operationsdisclosed herein. This apparatus may be specially constructed for therequired purposes, or it may be a general purpose computer selectivelyactivated or reconfigured by a computer program stored in the computer.Such a computer program may be stored in a computer readable storagemedium, such as, but not limited to, any type of disk including floppydisks, optical disks, CD-ROMs, and magnetic-optical disks, read-onlymemories (ROMs), random access memories (RAMs), EPROMs, EEPROMs,magnetic or optical cards, or any type of media suitable for storingelectronic instructions, each coupled to a computer system bus.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear as set forth in thedescription below. In addition, embodiments are not described withreference to any particular programming language. It will be appreciatedthat a variety of programming languages may be used to implement theteachings of the embodiments as described herein.

Embodiments may be provided as a computer program product, or software,that may include a machine-readable medium having stored thereoninstructions, which may be used to program a computer system (or otherelectronic devices) to perform a process according to the disclosedembodiments. A machine-readable medium includes any mechanism forstoring or transmitting information in a form readable by a machine(e.g., a computer). For example, a machine-readable (e.g.,computer-readable) medium includes a machine (e.g., a computer) readablestorage medium (e.g., read only memory (“ROM”), random access memory(“RAM”), magnetic disk storage media, optical storage media, flashmemory devices, etc.), a machine (e.g., computer) readable transmissionmedium (electrical, optical, acoustical), etc.

Any of the disclosed embodiments may be used alone or together with oneanother in any combination. Although various embodiments may have beenpartially motivated by deficiencies with conventional techniques andapproaches, some of which are described or alluded to within thespecification, the embodiments need not necessarily address or solve anyof these deficiencies, but rather, may address only some of thedeficiencies, address none of the deficiencies, or be directed towarddifferent deficiencies and problems where are not directly discussed.

In one embodiment, means for predictive query and latent structure queryimplementation and usage in a multi-tenant database system execute at anapplication in a computing device, a computing system, or a computingarchitecture, in which the application is enabled to communicate with aremote computing device over a public Internet, such as remote clients,thus establishing a cloud based computing service in which the clientsutilize the functionality of the remote application which implements thepredictive and latent structure query and usage capabilities.

FIG. 1 depicts an exemplary architecture 100 in accordance withdescribed embodiments.

In one embodiment, a production environment 111 is communicablyinterfaced with a plurality of client devices 106A-C through hostorganization 110. In one embodiment, a multi-tenant database system 130includes a relational data store 155, for example, to store datasets onbehalf of customer organizations 105A-C or users. The multi-tenantdatabase system 130 further stores indices for predictive queries 150,for instance, which are generated from datasets provided by, specifiedby, or stored on behalf of users and customer organizations 105A-C.

Multi-tenant database system 130 includes a plurality of underlyinghardware, software, and logic elements 120 that implement databasefunctionality and a code execution environment within the hostorganization 110. In accordance with one embodiment, multi-tenantdatabase system 130 implements the non-relational data store—andseparately implements a predictive database to store the indices forpredictive queries 150. The hardware, software, and logic elements 120of the multi-tenant database system 1230 are separate and distinct froma plurality of customer organizations (105A, 105B, and 105C) whichutilize the services provided by the host organization 110 bycommunicably interfacing to the host organization 110 via network 125.In such a way, host organization 110 may implement on-demand services,on-demand database services or cloud computing services to subscribingcustomer organizations 105A-C.

Host organization 110 receives input and other requests 115 from aplurality of customer organizations 105A-C via network 125 (such as apublic Internet). For example, the incoming PreQL queries, predictivequeries, API requests, or other input may be received from the customerorganizations 105A-C to be processed against the multi-tenant databasesystem 130.

In one embodiment, each customer organization 105A-C is an entityselected from the group consisting of: a separate and distinct remoteorganization, an organizational group within the host organization 110,a business partner of the host organization 110, or a customerorganization 105A-C that subscribes to cloud computing services providedby the host organization 110.

In one embodiment, requests 115 are received at, or submitted to, aweb-server 175 within host organization 110. Host organization 110 mayreceive a variety of requests for processing by the host organization110 and its multi-tenant database system 130. Incoming requests 115received at web-server 175 may specify which services from the hostorganization 110 are to be provided, such as query requests, searchrequest, status requests, database transactions, a processing request toretrieve, update, or store data on behalf of one of the customerorganizations 105A-C, and so forth. Web-server 175 may be responsiblefor receiving requests 115 from various customer organizations 105A-Cvia network 125 and provide a web-based interface to an end-user clientdevice 106A-C or machine originating such data requests 115.

Query interface 180 provides functionality to pass queries fromweb-server 175 into the multi-tenant database system 130 for executionagainst the indices for predictive queries 150 or the relational datastore 155. In one embodiment, the query interface 180 implements a PreQLApplication Programming Interface (API) or a JavaScript Object Notation(JSON) API interface through which queries may be executed against theindices for predictive queries 150 or the relational data store 155.Query optimizer 160 performs query translation and optimization, forinstance, on behalf of other functionality which possesses sufficientinformation to architect a query or PreQL query yet lacks the necessarylogic to actually construct the query syntax. Analysis engine 185operates to generate queryable indices for predictive queries fromtabular datasets or other data provided by, or specified by users.

Host organization 110 may implement a request interface 176 viaweb-server 175 or as a stand-alone interface to receive requests packetsor other requests 115 from the client devices 106A-C. Request interface176 further supports the return of response packets or other replies andresponses 116 in an outgoing direction from host organization 110 to theclient devices 106A-C. According to one embodiment, query interface 180implements a PreQL API interface and/or a JSON API interface withspecialized functionality to execute PreQL queries or other predictivequeries against the databases of the multi-tenant database system 130,such as the indices for predictive queries at element 150. For instance,query interface 180 may operate to query the predictive database withinhost organization 110 in fulfillment of such requests 115 from theclient devices 106A-C by issuing API calls with PreQL structured queryterms such as “PREDICT,” “RELATED,” “SIMILAR,” and “GROUP.” Alsoavailable are API calls for “UPLOAD” and “ANALYZE,” so as to upload newdata sets or define datasets to the predictive database 1350 and triggerthe analysis engine 185 to instantiate analysis of such data which inturn generates queryable indices in support of such queries.

FIG. 2 illustrates a block diagram of an example of an environment 210in which an on-demand database service might be used. Environment 210may include user systems 212, network 214, system 216, processor system217, application platform 218, network interface 220, tenant datastorage 222, system data storage 224, program code 226, and processspace 228. In other embodiments, environment 210 may not have all of thecomponents listed and/or may have other elements instead of, or inaddition to, those listed above.

Environment 210 is an environment in which an on-demand database serviceexists. User system 212 may be any machine or system that is used by auser to access a database user system. For example, any of user systems212 can be a handheld computing device, a mobile phone, a laptopcomputer, a work station, and/or a network of computing devices. Asillustrated in FIG. 2 (and in more detail in FIG. 3) user systems 212might interact via a network 214 with an on-demand database service,which is system 216.

An on-demand database service, such as system 216, is a database systemthat is made available to outside users that do not need to necessarilybe concerned with building and/or maintaining the database system, butinstead may be available for their use when the users need the databasesystem (e.g., on the demand of the users). Some on-demand databaseservices may store information from one or more tenants stored intotables of a common database image to form a multi-tenant database system(MTS). Accordingly, “on-demand database service 216” and “system 216” isused interchangeably herein. A database image may include one or moredatabase objects. A relational database management system (RDMS) or theequivalent may execute storage and retrieval of information against thedatabase object(s). Application platform 218 may be a framework thatallows the applications of system 216 to run, such as the hardwareand/or software, e.g., the operating system. In an embodiment, on-demanddatabase service 216 may include an application platform 218 thatenables creation, managing and executing one or more applicationsdeveloped by the provider of the on-demand database service, usersaccessing the on-demand database service via user systems 212, or thirdparty application developers accessing the on-demand database servicevia user systems 212.

The users of user systems 212 may differ in their respective capacities,and the capacity of a particular user system 212 might be entirelydetermined by permissions (permission levels) for the current user. Forexample, where a salesperson is using a particular user system 212 tointeract with system 216, that user system has the capacities allottedto that salesperson. However, while an administrator is using that usersystem to interact with system 216, that user system has the capacitiesallotted to that administrator. In systems with a hierarchical rolemodel, users at one permission level may have access to applications,data, and database information accessible by a lower permission leveluser, but may not have access to certain applications, databaseinformation, and data accessible by a user at a higher permission level.Thus, different users will have different capabilities with regard toaccessing and modifying application and database information, dependingon a user's security or permission level.

Network 214 is any network or combination of networks of devices thatcommunicate with one another. For example, network 214 can be any one orany combination of a LAN (local area network), WAN (wide area network),telephone network, wireless network, point-to-point network, starnetwork, token ring network, hub network, or other appropriateconfiguration. As the most common type of computer network in currentuse is a TCP/IP (Transfer Control Protocol and Internet Protocol)network, such as the global internetwork of networks often referred toas the “Internet” with a capital “I,” that network will be used in manyof the examples herein. However, it is understood that the networks thatthe claimed embodiments may utilize are not so limited, although TCP/IPis a frequently implemented protocol.

User systems 212 might communicate with system 216 using TCP/IP and, ata higher network level, use other common Internet protocols tocommunicate, such as HTTP, FTP, AFS, WAP, etc. In an example where HTTPis used, user system 212 might include an HTTP client commonly referredto as a “browser” for sending and receiving HTTP messages to and from anHTTP server at system 216. Such an HTTP server might be implemented asthe sole network interface between system 216 and network 214, but othertechniques might be used as well or instead. In some implementations,the interface between system 216 and network 214 includes load sharingfunctionality, such as round-robin HTTP request distributors to balanceloads and distribute incoming HTTP requests evenly over a plurality ofservers. At least as for the users that are accessing that server, eachof the plurality of servers has access to the MTS' data; however, otheralternative configurations may be used instead.

In one embodiment, system 216, shown in FIG. 2, implements a web-basedcustomer relationship management (CRM) system. For example, in oneembodiment, system 216 includes application servers configured toimplement and execute CRM software applications as well as providerelated data, code, forms, webpages and other information to and fromuser systems 212 and to store to, and retrieve from, a database systemrelated data, objects, and Webpage content. With a multi-tenant system,data for multiple tenants may be stored in the same physical databaseobject, however, tenant data typically is arranged so that data of onetenant is kept logically separate from that of other tenants so that onetenant does not have access to another tenant's data, unless such datais expressly shared. In certain embodiments, system 216 implementsapplications other than, or in addition to, a CRM application. Forexample, system 216 may provide tenant access to multiple hosted(standard and custom) applications, including a CRM application. User(or third party developer) applications, which may or may not includeCRM, may be supported by the application platform 218, which managescreation, storage of the applications into one or more database objectsand executing of the applications in a virtual machine in the processspace of the system 216.

One arrangement for elements of system 216 is shown in FIG. 2, includinga network interface 220, application platform 218, tenant data storage222 for tenant data 223, system data storage 224 for system data 225accessible to system 216 and possibly multiple tenants, program code 226for implementing various functions of system 216, and a process space228 for executing MTS system processes and tenant-specific processes,such as running applications as part of an application hosting service.Additional processes that may execute on system 216 include databaseindexing processes.

Several elements in the system shown in FIG. 2 include conventional,well-known elements that are explained only briefly here. For example,each user system 212 may include a desktop personal computer,workstation, laptop, PDA, cell phone, or any wireless access protocol(WAP) enabled device or any other computing device capable ofinterfacing directly or indirectly to the Internet or other networkconnection. User system 212 typically runs an HTTP client, e.g., abrowsing program, such as Microsoft's Internet Explorer browser, aMozilla or Firefox browser, an Opera, or a WAP-enabled browser in thecase of a smartphone, tablet, PDA or other wireless device, or the like,allowing a user (e.g., subscriber of the multi-tenant database system)of user system 212 to access, process and view information, pages andapplications available to it from system 216 over network 214. Each usersystem 212 also typically includes one or more user interface devices,such as a keyboard, a mouse, trackball, touch pad, touch screen, pen orthe like, for interacting with a graphical user interface (GUI) providedby the browser on a display (e.g., a monitor screen, LCD display, etc.)in conjunction with pages, forms, applications and other informationprovided by system 216 or other systems or servers. For example, theuser interface device can be used to access data and applications hostedby system 216, and to perform searches on stored data, and otherwiseallow a user to interact with various GUI pages that may be presented toa user. As discussed above, embodiments are suitable for use with theInternet, which refers to a specific global internetwork of networks.However, it is understood that other networks can be used instead of theInternet, such as an intranet, an extranet, a virtual private network(VPN), a non-TCP/IP based network, any LAN or WAN or the like.

According to one embodiment, each user system 212 and all of itscomponents are operator configurable using applications, such as abrowser, including computer code run using a central processing unitsuch as an Intel Pentium® processor or the like. Similarly, system 216(and additional instances of an MTS, where more than one is present) andall of their components might be operator configurable usingapplication(s) including computer code to run using a central processingunit such as processor system 217, which may include an Intel Pentium®processor or the like, and/or multiple processor units.

According to one embodiment, each system 216 is configured to providewebpages, forms, applications, data and media content to user (client)systems 212 to support the access by user systems 212 as tenants ofsystem 216. As such, system 216 provides security mechanisms to keepeach tenant's data separate unless the data is shared. If more than oneMTS is used, they may be located in close proximity to one another(e.g., in a server farm located in a single building or campus), or theymay be distributed at locations remote from one another (e.g., one ormore servers located in city A and one or more servers located in cityB). As used herein, each MTS may include one or more logically and/orphysically connected servers distributed locally or across one or moregeographic locations. Additionally, the term “server” is meant toinclude a computer system, including processing hardware and processspace(s), and an associated storage system and database application(e.g., OODBMS or RDBMS) as is well known in the art. It is understoodthat “server system” and “server” are often used interchangeably herein.Similarly, the database object described herein can be implemented assingle databases, a distributed database, a collection of distributeddatabases, a database with redundant online or offline backups or otherredundancies, etc., and might include a distributed database or storagenetwork and associated processing intelligence.

FIG. 3 illustrates a block diagram of an embodiment of elements of FIG.2 and various possible interconnections between these elements. FIG. 3also illustrates environment 210. However, in FIG. 3, the elements ofsystem 216 and various interconnections in an embodiment are furtherillustrated. FIG. 3 shows that user system 212 may include a processorsystem 212A, memory system 212B, input system 212C, and output system212D. FIG. 3 shows network 214 and system 216. FIG. 3 also shows thatsystem 216 may include tenant data storage 222, tenant data 223, systemdata storage 224, system data 225, User Interface (UI) 330, ApplicationProgram Interface (API) 332 (e.g., a PreQL or JSON API), PL/SOQL 334,save routines 336, application setup mechanism 338, applications servers300 ₁-300 _(N), system process space 302, tenant process spaces 304,tenant management process space 310, tenant storage area 312, userstorage 314, and application metadata 316. In other embodiments,environment 210 may not have the same elements as those listed aboveand/or may have other elements instead of, or in addition to, thoselisted above.

User system 212, network 214, system 216, tenant data storage 222, andsystem data storage 224 were discussed above in FIG. 2. As shown by FIG.3, system 216 may include a network interface 220 (of FIG. 2)implemented as a set of HTTP application servers 300, an applicationplatform 218, tenant data storage 222, and system data storage 224. Alsoshown is system process space 302, including individual tenant processspaces 304 and a tenant management process space 310. Each applicationserver 300 may be configured to tenant data storage 222 and the tenantdata 223 therein, and system data storage 224 and the system data 225therein to serve requests of user systems 212. The tenant data 223 mightbe divided into individual tenant storage areas 312, which can be eithera physical arrangement and/or a logical arrangement of data. Within eachtenant storage area 312, user storage 314 and application metadata 316might be similarly allocated for each user. For example, a copy of auser's most recently used (MRU) items might be stored to user storage314. Similarly, a copy of MRU items for an entire organization that is atenant might be stored to tenant storage area 312. A UI 330 provides auser interface and an API 332 (e.g., a PreQL or JSON API) provides anapplication programmer interface to system 216 resident processes tousers and/or developers at user systems 212. The tenant data and thesystem data may be stored in various databases, such as one or moreOracle™ databases.

Application platform 218 includes an application setup mechanism 338that supports application developers' creation and management ofapplications, which may be saved as metadata into tenant data storage222 by save routines 336 for execution by subscribers as one or moretenant process spaces 304 managed by tenant management process space 310for example. Invocations to such applications may be coded using PL/SOQL334 that provides a programming language style interface extension toAPI 332 (e.g., a PreQL or JSON API). Invocations to applications may bedetected by one or more system processes, which manages retrievingapplication metadata 316 for the subscriber making the invocation andexecuting the metadata as an application in a virtual machine.

Each application server 300 may be communicably coupled to databasesystems, e.g., having access to system data 225 and tenant data 223, viaa different network connection. For example, one application server 300₁ might be coupled via the network 214 (e.g., the Internet), anotherapplication server 300 _(N-1) might be coupled via a direct networklink, and another application server 300 _(N) might be coupled by yet adifferent network connection. Transfer Control Protocol and InternetProtocol (TCP/IP) are typical protocols for communicating betweenapplication servers 300 and the database system. However, it will beapparent to one skilled in the art that other transport protocols may beused to optimize the system depending on the network interconnect used.

In certain embodiments, each application server 300 is configured tohandle requests for any user associated with any organization that is atenant. Because it is desirable to be able to add and remove applicationservers from the server pool at any time for any reason, there ispreferably no server affinity for a user and/or organization to aspecific application server 300. In one embodiment, therefore, aninterface system implementing a load balancing function (e.g., an F5Big-IP load balancer) is communicably coupled between the applicationservers 300 and the user systems 212 to distribute requests to theapplication servers 300. In one embodiment, the load balancer uses aleast connections algorithm to route user requests to the applicationservers 300. Other examples of load balancing algorithms, such as roundrobin and observed response time, also can be used. For example, incertain embodiments, three consecutive requests from the same user mayhit three different application servers 300, and three requests fromdifferent users may hit the same application server 300. In this manner,system 216 is multi-tenant, in which system 216 handles storage of, andaccess to, different objects, data and applications across disparateusers and organizations.

As an example of storage, one tenant might be a company that employs asales force where each salesperson uses system 216 to manage their salesprocess. Thus, a user might maintain contact data, leads data, customerfollow-up data, performance data, goals and progress data, etc., allapplicable to that user's personal sales process (e.g., in tenant datastorage 222). In an example of a MTS arrangement, since all of the dataand the applications to access, view, modify, report, transmit,calculate, etc., can be maintained and accessed by a user system havingnothing more than network access, the user can manage his or her salesefforts and cycles from any of many different user systems. For example,if a salesperson is visiting a customer and the customer has Internetaccess in their lobby, the salesperson can obtain critical updates as tothat customer while waiting for the customer to arrive in the lobby.

While each user's data might be separate from other users' dataregardless of the employers of each user, some data might beorganization-wide data shared or accessible by a plurality of users orall of the users for a given organization that is a tenant. Thus, theremight be some data structures managed by system 216 that are allocatedat the tenant level while other data structures might be managed at theuser level. Because an MTS might support multiple tenants includingpossible competitors, the MTS may have security protocols that keepdata, applications, and application use separate. Also, because manytenants may opt for access to an MTS rather than maintain their ownsystem, redundancy, up-time, and backup are additional functions thatmay be implemented in the MTS. In addition to user-specific data andtenant specific data, system 216 might also maintain system level datausable by multiple tenants or other data. Such system level data mightinclude industry reports, news, postings, and the like that are sharableamong tenants.

In certain embodiments, user systems 212 (which may be client systems)communicate with application servers 300 to request and updatesystem-level and tenant-level data from system 216 that may requiresending one or more queries to tenant data storage 222 and/or systemdata storage 224. System 216 (e.g., an application server 300 in system216) automatically generates one or more SQL statements or PreQLstatements (e.g., one or more SQL or PreQL queries respectively) thatare designed to access the desired information. System data storage 224may generate query plans to access the requested data from the database.

Each database can generally be viewed as a collection of objects, suchas a set of logical tables, containing data fitted into predefinedcategories. A “table” is one representation of a data object, and may beused herein to simplify the conceptual description of objects and customobjects as described herein. It is understood that “table” and “object”may be used interchangeably herein. Each table generally contains one ormore data categories logically arranged as columns or fields in aviewable schema. Each row or record of a table contains an instance ofdata for each category defined by the fields. For example, a CRMdatabase may include a table that describes a customer with fields forbasic contact information such as name, address, phone number, faxnumber, etc. Another table might describe a purchase order, includingfields for information such as customer, product, sale price, date, etc.In some multi-tenant database systems, standard entity tables might beprovided for use by all tenants. For CRM database applications, suchstandard entities might include tables for Account, Contact, Lead, andOpportunity data, each containing pre-defined fields. It is understoodthat the word “entity” may also be used interchangeably herein with“object” and “table.”

In some multi-tenant database systems, tenants may be allowed to createand store custom objects, or they may be allowed to customize standardentities or objects, for example by creating custom fields for standardobjects, including custom index fields. In certain embodiments, forexample, all custom entity data rows are stored in a single multi-tenantphysical table, which may contain multiple logical tables perorganization. It is transparent to customers that their multiple“tables” are in fact stored in one large table or that their data may bestored in the same table as the data of other customers.

FIG. 4 illustrates a diagrammatic representation of a machine 400 in theexemplary form of a computer system, in accordance with one embodiment,within which a set of instructions, for causing the machine/computersystem 400 to perform any one or more of the methodologies discussedherein, may be executed. In alternative embodiments, the machine may beconnected (e.g., networked) to other machines in a Local Area Network(LAN), an intranet, an extranet, or the public Internet. The machine mayoperate in the capacity of a server or a client machine in aclient-server network environment, as a peer machine in a peer-to-peer(or distributed) network environment, as a server or series of serverswithin an on-demand service environment. Certain embodiments of themachine may be in the form of a personal computer (PC), a tablet PC, aset-top box (STB), a Personal Digital Assistant (PDA), a cellulartelephone, a web appliance, a server, a network router, switch orbridge, computing system, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine. Further, while only a single machine is illustrated,the term “machine” shall also be taken to include any collection ofmachines (e.g., computers) that individually or jointly execute a set(or multiple sets) of instructions to perform any one or more of themethodologies discussed herein.

The exemplary computer system 400 includes a processor 402, a mainmemory 404 (e.g., read-only memory (ROM), flash memory, dynamic randomaccess memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM(RDRAM), etc., static memory such as flash memory, static random accessmemory (SRAM), volatile but high-data rate RAM, etc.), and a secondarymemory 418 (e.g., a persistent storage device including hard disk drivesand a persistent database and/or a multi-tenant databaseimplementation), which communicate with each other via a bus 430. Mainmemory 404 includes stored indices 424, an analysis engine 423, and aPreQL API 425. Main memory 404 and its sub-elements are operable inconjunction with processing logic 426 and processor 402 to perform themethodologies discussed herein. The computer system 400 may additionallyor alternatively embody the server side elements as described above.

Processor 402 represents one or more general-purpose processing devicessuch as a microprocessor, central processing unit, or the like. Moreparticularly, the processor 402 may be a complex instruction setcomputing (CISC) microprocessor, reduced instruction set computing(RISC) microprocessor, very long instruction word (VLIW) microprocessor,processor implementing other instruction sets, or processorsimplementing a combination of instruction sets. Processor 402 may alsobe one or more special-purpose processing devices such as an applicationspecific integrated circuit (ASIC), a field programmable gate array(FPGA), a digital signal processor (DSP), network processor, or thelike. Processor 402 is configured to execute the processing logic 426for performing the operations and functionality which is discussedherein.

The computer system 400 may further include a network interface card408. The computer system 400 also may include a user interface 410 (suchas a video display unit, a liquid crystal display (LCD), or a cathoderay tube (CRT)), an alphanumeric input device 412 (e.g., a keyboard), acursor control device 414 (e.g., a mouse), and a signal generationdevice 416 (e.g., an integrated speaker). The computer system 400 mayfurther include peripheral device 436 (e.g., wireless or wiredcommunication devices, memory devices, storage devices, audio processingdevices, video processing devices, etc.).

The secondary memory 418 may include a non-transitory machine-readableor computer readable storage medium 431 on which is stored one or moresets of instructions (e.g., software 422) embodying any one or more ofthe methodologies or functions described herein. The software 422 mayalso reside, completely or at least partially, within the main memory404 and/or within the processor 402 during execution thereof by thecomputer system 400, the main memory 404 and the processor 402 alsoconstituting machine-readable storage media. The software 422 mayfurther be transmitted or received over a network 420 via the networkinterface card 408.

FIG. 5A depicts a tablet computing device 501 and a hand-held smartphone502 each having a circuitry integrated therein as described inaccordance with the embodiments. As depicted, each of the tabletcomputing device 501 and the hand-held smartphone 502 include atouchscreen interface 503 and an integrated processor 504 in accordancewith disclosed embodiments.

For example, in one embodiment, a system embodies a tablet computingdevice 501 or a hand-held smartphone 502, in which a display unit of thesystem includes a touchscreen interface 503 for the tablet or thesmartphone and further in which memory and an integrated circuitoperating as an integrated processor are incorporated into the tablet orsmartphone, in which the integrated processor implements one or more ofthe embodiments described herein for use of a predictive and latentstructure query implementation through an on-demand and/or multi-tenantdatabase system such as a cloud computing service provided via a publicInternet as a subscription service. In one embodiment, the integratedcircuit described above or the depicted integrated processor of thetablet or smartphone is an integrated silicon processor functioning as acentral processing unit (CPU) and/or a Graphics Processing Unit (GPU)for a tablet computing device or a smartphone.

Although the tablet computing device 501 and hand-held smartphone 502may have limited processing capabilities, each is nevertheless enabledto utilize the predictive and latent structure query capabilitiesprovided by a host organization as a cloud based service, for instance,such as host organization 110 depicted at FIG. 1.

FIG. 5B is a block diagram 500 of an embodiment of tablet computingdevice 501, hand-held smartphone 502, or other mobile device in whichtouchscreen interface connectors are used. Processor 510 performs theprimary processing operations. Audio subsystem 520 represents hardware(e.g., audio hardware and audio circuits) and software (e.g., drivers,codecs) components associated with providing audio functions to thecomputing device. In one embodiment, a user interacts with the tabletcomputing device or smart phone by providing audio commands that arereceived and processed by processor 510.

Display subsystem 530 represents hardware (e.g., display devices) andsoftware (e.g., drivers) components that provide a visual and/or tactiledisplay for a user to interact with the tablet computing device or smartphone. Display subsystem 530 includes display interface 532, whichincludes the particular screen or hardware device used to provide adisplay to a user. In one embodiment, display subsystem 530 includes atouchscreen device that provides both output and input to a user.

I/O controller 540 represents hardware devices and software componentsrelated to interaction with a user. I/O controller 540 can operate tomanage hardware that is part of audio subsystem 520 and/or displaysubsystem 530. Additionally, I/O controller 540 illustrates a connectionpoint for additional devices that connect to the tablet computing deviceor smart phone through which a user might interact. In one embodiment,I/O controller 540 manages devices such as accelerometers, cameras,light sensors or other environmental sensors, or other hardware that canbe included in the tablet computing device or smart phone. The input canbe part of direct user interaction, as well as providing environmentalinput to the tablet computing device or smart phone.

In one embodiment, the tablet computing device or smart phone includespower management 550 that manages battery power usage, charging of thebattery, and features related to power saving operation. Memorysubsystem 560 includes memory devices for storing information in thetablet computing device or smart phone. Connectivity 570 includeshardware devices (e.g., wireless and/or wired connectors andcommunication hardware) and software components (e.g., drivers, protocolstacks) to the tablet computing device or smart phone to communicatewith external devices. Cellular connectivity 572 may include, forexample, wireless carriers such as GSM (global system for mobilecommunications), CDMA (code division multiple access), TDM (timedivision multiplexing), or other cellular service standards). Wirelessconnectivity 574 may include, for example, activity that is notcellular, such as personal area networks (e.g., Bluetooth), local areanetworks (e.g., WiFi), and/or wide area networks (e.g., WiMax), or otherwireless communication.

Peripheral connections 580 include hardware interfaces and connectors,as well as software components (e.g., drivers, protocol stacks) to makeperipheral connections as a peripheral device (“to” 582) to othercomputing devices, as well as have peripheral devices (“from” 584)connected to the tablet computing device or smart phone, including, forexample, a “docking” connector to connect with other computing devices.Peripheral connections 580 include common or standards-based connectors,such as a Universal Serial Bus (USB) connector, DisplayPort includingMiniDisplayPort (MDP), High Definition Multimedia Interface (HDMI),Firewire, etc.

FIG. 6 depicts a simplified flow for probabilistic modeling.Probabilistic modeling requires a series of choices and assumptions. Forinstance, it is possible to trade off fidelity and detail withtractability. Assumptions define an outcome space which may beconsidered hypotheses, and in the modeling view, one of these possiblehypotheses actually occurs.

For instance, at element 601 the probabilistic modeling flow depictsassumptions which leverage prior knowledge 605. The flow advances toelement 602 where there is a hypothesis space which defines a space ofpossible outcomes 606. The probabilistic modeling flow advances toelement 603 which results in hidden structure based on learning 607derived from the defined space of possible outcomes 606. The flow thenadvances to element 604 where observed data is utilized by gatheringinformation from available sources 608 which then loops back to learningat element 607 to recursively better inform the probabilistic model.

The hidden structure at 603 is used to generate data. The hiddenstructure 603 and the resulting generated data may be considered thegenerative view. Learning 607 uses available sources of information andinferences about the hidden structure which may include certain modelingassumptions (“prior”), as well as data observed (“likelihood”), fromwhich a combination of prior and likelihood may be utilized to drawconclusions (“posterior”).

Such assumptions yield hypothesis space and additionally provide a meansby which probabilities may be assigned to such assumptions, thusyielding a probability distribution on hypotheses, given actual dataobserved.

The modeling assumptions implemented by the analysis engine to generatequeryable indices define both a hypothesis space as well as a recipe forassigning a probability to each hypothesis given some data. Aprobability distribution thus results in which each hypothesis is anoutcome, for which there can be a great many available and possibleoutcomes, each with varying probability. There can also be a great manyhypotheses and finding the best ones to explain the data is not astraight forward or obvious proposition.

Probabilistic inference thus presents the problem of how to searchthrough the available hypothesis space to find the ones that give thebest explanations for the data at hand. The analysis engine describedherein implements a range of methods including functionality to solvethe math directly, functionality to leverage optimization to find thepeak of the hypothesis space, and functionality to implement randomwalks through the hypothesis space.

The probabilistic modeling makes assumptions 601 and using theassumptions, a hypothesis space 602 is defined. Probabilities areassigned to the hypotheses given data observed and then inference isused to figure out which of those explanatory hypotheses are plausibleand which one is the best.

FIG. 7 illustrates an exemplary landscape upon which a random walk maybe performed. Experts in the field do not agree on how to select thebest hypothesis but there are several favored approaches. In simplecases, functionality can use math to solve the equations directly. Otheroptimization methods are popular such as hill climbing and itsrelatives. In certain described embodiments, the analysis engineutilizes Monte Carlo methods in which a random walk is taken through thespace of hypotheses. Random does not mean inefficient or stupidlynavigating without aim, direction, or purpose. In fact, efficientlynavigating these huge spaces is a one of the innovations utilized by theanalysis engine to improve the path taken by a random walk.

Consider the landscape of the hypothesis space 703 through which arandom walk may be performed in which each axis is one dimension in thehypothesis space 703. On the vertical axis at element 701 hidden value 2is represented and the horizontal axis at element 702, hidden value 1 isrepresented. Real spaces can have many dimensions, far more than the twodimensions shown here for the sake of simplicity. Height of the surfaceformed by the random walk method is the probability of the hiddenvariables, given data and modeling assumptions.

Exploration starts by taking a random step somewhere, anywhere, and ifthe step is higher then it is kept, but if the step is lower, then it issometimes kept and other times it is not, electing to stay put instead.The result is extremely useful as it is guaranteed to explore the spacein proportion to the true probability values. Over the long run twopeaks result as can be seen in example provided, one corresponding toeach of the provided dimensions (e.g., 701 and 702 at the two axesdepicted). Conversely, simple hill climbing will get caught at the topof one hill and fail to yield the distinct peaks. Such an approach thusexplores the whole of the hypothesis space whereas conventionaltechniques will not. Other innovations include added intelligence aboutjumps as well as functionality for exploring one or many dimensions at atime.

FIG. 8 depicts an exemplary tabular dataset. With tabular data, each rowcontains information about one particular entity and each of the manyrows are independent from one another. Each column contains a singletype of information, and such data may be data typed as, for example,numerical, categorical, Boolean, etc. Column types may be mixed andmatched within a table and the data type applied or assigned for anygiven column is uniform amongst all cells or fields within the entirecolumn, but one column's data type does not restrict any particular datatype on any other column. Such tabular data is therefore a very goodmatch to a single database table of a relational database which providesa tabular dataset. The tabular data is also a good match to a dataframein “R.”

In the exemplary table depicted, element 802 forms entities, each of therows being mammals and at element 801, each of the columns are features,characteristics, or variables that describe the mammals. Most of thecolumns are data-typed as Boolean but some are categorical.

Note that element 804 depicts an observed cell, that is to say, data isprovided for that cell in contrast to element 803 which is an unobservedcell for which there is no data available. The unobserved cells 803 thusare null values whereas observed cells have data populated in the field,whether that data is Boolean, categorical, a value, an enumeratedelement, or whatever is appropriate for the data type of the column. Allof the cells depicted as white or blank are unobserved cells.

A co-assignment matrix for dimensions, where:

C_(ij)=Pr[z_(i)=z_(j)]

results in the probability that dimensions i and j share a common causeand therefore are modeled by the same Dirichlet process mixture. Labelsshow the consensus dimension groups (probability >0:75). These reflectattributes that share a common cause and thus co-vary, while theremainder of the matrix captures correlations between these discoveredcauses, for instance, mammals rarely have feathers or fly, ungulates arenot predators, and so forth. Each dimension group picks out a differentcross-cutting categorization of the rows (e.g. vertebrates, birds,canines, etc.).

FIG. 9 depicts means for deriving motivation or causal relationshipsbetween observed data, such as the data provided in tabular form at FIG.8. In the exemplary data about mammals and their characteristics, it maybe expected that some causal relationships can be appropriately derived.Conversely, if the tabular data is modified such that the price of teain China is provided, such data, although present and observed,intuitively does not in any way help or hurt the resultant predictionsmade about mammals based on the observed data. Such extraneous data(e.g., the price of tea in China within a table describing mammals)represents noise and needs to be accommodated because real-world data isvery often noisy and poorly structured. The analysis engine needs tofind the appropriate motivation for its predictions and not be misled bynoisy irrelevant data, despite such data being actually “observed”within the provided dataset. Real-world data simply is not pristine andthus presents a very real problem if a scalable solution is to beutilized which renders appropriate predictions without requiring customsolutions to be developed manually for each and every dataset presented.The analysis engine must therefore employ models which understand thatsome data simply does not matter to a given hypothesis or predictiverelationship. For instance, some columns may not matter or certaincolumns may carry redundant information. Some columns may therefore bepredictively related and may thus be grouped together whereas others arenot predictively related, and as such, are grouped separately. Thesegroups of columns are referred to as “views.”

Two distinct views are depicted. View 1 at element 905 resulting fromcasual process 1 (element 903) and view 2 at element 906 resulting fromcasual process 2 (element 904). Within each view 905 and 906, the rowsare grouped into categories. As shown, view 1 corresponds to features1-3 at elements 907, 908, and 909 and view 2 corresponds to features 4-5at elements 910 and 911. Each of the “features” of the respective viewscorresponding to columns of the tabular dataset depicted at FIG. 8 whichin the example provided, define characteristics, variables, or featuresabout the respective mammals listed as entities (e.g., rows).

Entities 1-2 are then depicted at elements 901 and 902 and within theviews the respective cell or field values are then depicted. Notably,the analysis engine has identified two column groupings, specifically,views 1 and 2 at elements 905 and 906, and thus, different predictiverelationships may be identified which are tailored to the particularviews.

FIG. 10A depicts an exemplary cross-categorization in still furtherdetail. Utilizing cross-categorization, columns/features are groupedinto views and rows/entities are grouped into categories. Views 1-3 aredepicted here in which view 1 at element 1001 has 12 features, view 2 atelement 1002 has 10 features, and view 3 at element 1003 has 8 features.Again, the features of the respective views correspond to columns fromthe tabular dataset provided. At view 1 (element 1001) it can be seenthat three entities are provided within the three different categoriesof the view. Entity 1 and 2 at elements 1005 and 1006 are both withinthe topmost category of view 1, entity 3 at element 1007 is within themiddle category, and none of the specifically listed entities (e.g.,rows) appear within the bottom category. The blacked out rows representthe entities 1-3 (1005, 1006, 1007) and as can be seen at view 2(element 1002) the arrangement changes. At view 2 there are only 10features and just one category which possesses all three of the listedentities (rows) 1005, 1006, and 1007. Then moving to view 3 at element1003, there are four categories and each of the three blacked outentities (rows) 1005, 1006, and 1007 reside within distinct categories.

Element 1004 provides a zoomed in depiction of view 3, the same aselement 1003 but with additional detail depicted. At element 1004 it canthus be seen that each of the categories possesses multiple entities,each with the actual data points corresponding to the cell values infrom the table for the columns actually listed by the categories of view3 at element 1004. For instance, category 1 has 16 total entities,category 2 has 8 entities, category 3 has 4 entities, and category 4 hastwo entities. Category 3 is then zoomed in still further such that itcan be seen which data elements are observed cells 1008 (marked with“X”) vs. unobserved cells 1009 (e.g., the blanks representing nullvalues, missing data, unknown data, etc.).

A single cross-categorization is a particular way of slicing and dicingthe table or dataset of tabular data. First by column and then by row,providing a particular kind of process to yield a desired structuredspace. A probability is then assigned to each cross-categorization thusresulting in probability distributions. More complexcross-categorizations yielding more views and more categories arefeasible but are in actuality less probable in and of themselves and aretherefore typically warranted only when the underlying data reallysupports them. The more complex cross-categorizations are supported butare not utilized by default.

Probabilistic modeling using clustering techniques, including inferencein Dirichlet process mixture models, present difficulty when differentdimensions are best explained by very different clusterings.Nevertheless, embodiments of the analysis engine described hereinovercome such difficulties through an inference method whichautomatically discovers the number of independent nonparametric Bayesianmodels needed to explain the data, using a separate Dirichlet processmixture model for each group in an inferred partition of the dimensions.Unlike a Dirichlet Process mixture (DP mixture), the describedimplementation is exchangeable over both the rows of a heterogeneousdata array (the samples) and the columns (new dimensions), and cantherefore model any dataset as the number of samples and dimensions bothgo to infinity. Efficiency and robustness is improved through use ofalgorithms described which in certain instances require no preprocessingto identify veridical causal structure provided in raw datasets.

Clustering techniques are widely used in data analysis for problems ofsegmentation in industry, exploratory analysis in science, and as apreprocessing step to improve performance of further processing indistributed computing and in data compression. However, as datasets growlarger and noisier, the assumption that a single clustering ordistribution over clusterings can account for all the variability in theobservations becomes less realistic if not wholly infeasible.

From a machine learning perspective, this is an unsupervised version ofthe feature selection problem: different subsets of measurements will,in general, induce different natural clusterings of the data. From acognitive science and artificial intelligence perspective, this issue isreflected in work that seeks multiple representations of data instead ofa single monolithic representation.

As a limiting case, a robust clustering method is able to ignore aninfinite number of uniformly random or perfectly deterministicmeasurements. The assumption that a single nonparametric model mustexplain all the dimensions is partly responsible for the accuracy issuesa Dirichlet Process mixture often encounters in high dimensionalsettings. Dirichlet Process mixture based classifiers via classconditional density estimation highlight the problem. For instance,while a discriminative classifier can assign low weight to noisy ordeterministic and therefore irrelevant dimensions, a generative modelmust explain them. If there are enough irrelevancies, it ignores thedimensions relevant to classification in the process. Combined with slowMCMC convergence, these difficulties have inhibited the use ofnonparametric Bayesian methods in many applications.

To overcome these limitations, an unsupervised cross-categorizationlearning technique is utilized for clustering based on MCMC inference ina novel nested nonparametric Bayesian model. This model can be viewed asa Dirichlet Process mixture over the dimensions or columns of Dirichletprocess mixture models over sampled data points or rows. Conditioned ona partition of the dimensions, the analysis engine's model reduces to anindependent product of DP mixtures, but the partition of the dimensions,and therefore the number and domain of independent nonparametricBayesian models, is also inferred from the data.

Standard feature selection results in the case where the partition ofdimensions has only two groups. The described model utilizes an MCMCapproach because both model selection and deterministic approximationsseem intractable due to the combinatorial explosion of latent variables,with changing numbers of latent variables as the partition of thedimensions changes.

The hypothesis space captured by the described model issuper-exponentially larger than that of a Dirichlet process mixture,with a very different structure than a Hierarchical Dirichlet Process. Agenerative process, viewed as a model for heterogeneous data arrays withN rows, D columns of fixed type and values missing at random, can bedescribed as follows:

1. For dεD:

-   -   (a) Generate hyperparameters {right arrow over (λ_(d))} from an        appropriate hyper-prior.    -   (b) Generate the model assignment z_(d) for dimension d from a        Chinese restaurant process with hyperparameter α (with α from a        vague hyperprior).

2. For each group g in the dimension partition {z_(d)}:

-   -   (a) Fr each sampled datapoint (or row) rεR, generate a cluster        assignment z_(r) ^(g) from a Chinese restaurant process with        hyperparameter α_(g) (with α_(g) from a vague hyperprior).    -   (b) For each cluster e in the row partition for this group of        dimensions {z_(d) ^(g)}:        -   i. For each dimension d, generate component model parameters            {right arrow over (θ_(e) ^(d))} from an appropriate prior            and {right arrow over (λ_(d))}.        -   ii. For each data cell x_((r,d)) in this component (z_(r)            ^(g) ^(d) =c for dεD), generate its value from an            appropriate likelihood and {right arrow over (θ_(e) ^(d))}.

In probability theory, the Chinese restaurant process is a discrete-timestochastic process, whose value at any positive-integer time n is apartition B_(n), of the set {1, 2, 3, . . . , n} whose probabilitydistribution is determined as follows: At time n=1, the trivialpartition {{1}} is obtained with probability 1 and at time n+1 theelement n+1 is either: (a) added to one of the blocks of the partitionB_(n), where each block is chosen with probability |b|/(n+1) where |b|is the size of the block, or alternatively (b) added to the partitionB_(n) as a new singleton block, with probability 1/(n+1). The randompartition so generated is exchangeable in the sense that relabeling {1,n} does not change the distribution of the partition, and it isconsistent in the sense that the law of the partition of n−1 obtained byremoving the element n from the random partition at time n is the sameas the law of the random partition at time n−1.

The model encodes a very different inductive bias than the Indian buffetprocess (IBP) adaptation of the Chinese restaurant process, discoveringindependent systems of categories over heterogeneous data vectors, asopposed to features that are typically additively combined. It is alsoinstructive to contrast the asymptotic capacity of the model with thatof a Dirichlet Process mixture. The Dirichlet Process mixture hasarbitrarily large asymptotic capacity as the number of samples goes toinfinity. Stated differently, the Dirichlet Process mixture can modelany distribution over finite dimensional vectors given enough data.However, if the number of dimensions (or features) is taken to infinity,it is no longer asymptotically consistent. That is, if a sequence ofdatasets is generated by sampling the first K₁ dimensions from a mixtureand then append K₂>>K₁ dimensions that are constant valued (e.g. theprice of tea in China), it will eventually be forced to model only thosedimensions, ignoring the statistical structure in the first K₁. Incontrast, the model implemented via the analysis engine according to thedescribed embodiments has asymptotic capacity both in terms of thenumber of samples and the number of dimensions, and is infinitelyexchangeable with respect to both quantities.

As a consequence, the model implemented via the analysis engine isself-consistent over the subset of variables measured, and can thusenjoy considerable robustness in the face of noisy, missing, andirrelevant measurements or confounding statistical signals. This isespecially helpful in demographic settings and in high-throughputbiology, where noisy, or coherently co-varying but orthogonal,measurements are the norm, and in which each data vector arises frommultiple, independent, generative processes in the real-world.

The algorithm and model implemented via the analysis engine builds upona general-purpose MCMC algorithm for probabilistic programs scalinglinearly per iteration in the number of rows and columns and includinginference over all hyperparameters.

FIG. 10B depicts an assessment of convergence, showing inferred versusground truth providing joint score for greater than 1000 MCMC runs (200iterations each) with varying dataset sizes (up to 512 by 512, requiring1-10 minutes each) and true dimension groups. A strong majority ofpoints fall near the ground truth dashed line, indicating reasonableconvergence; perfect linearity is not expected, partly due to posterioruncertainty.

Massively parallel implementations exploit the conditionalindependencies in the described model. Because the described method isessentially parameter free (e.g. with improper uniform hyperpriors),robust to noisy and/or irrelevant measurements generated by multipleinteracting causes, and supports arbitrarily sparsely observed,heterogeneous data, it may be broadly applicable in exploratory dataanalysis. Additionally, the performance of the utilized MCMC algorithmsuggests that the described approach to nesting latent variable modelsin a Dirichlet process over dimensions may be applied to generaterobust, rapidly converging, cross-cutting variants of a wide variety ofnonparametric Bayesian techniques.

The predictive and latent structure query capability and associated APIsmake use of a predictive database that finds the causes behind data anduses these causes to predict and explain the future in a highlyautomated fashion heretofore unavailable, thus allowing any developer tocarry out scientific inquires against a dataset without requiring customprogramming and consultation with mathematicians and other such experts.Such causes are revealed by latent structure and relationships learnedby the analysis engine.

The predictive and latent structure query capability works by searchingthrough the massive hypothesis space of all possible relationshipspresent in a dataset, using an advanced Bayesian machine learningalgorithm and thus offers developers: state of the art inferenceperformance and predictive accuracy on a very wide range of real-worlddatasets, with no manual parameter tuning whatsoever; scalability tovery large datasets, including very high-dimensional data with hundredsof thousands or millions of columns or rows; completely flexiblepredictions (e.g., able to predict the value of any subset of columns,given values for any other subset) without any retraining or adjustmentas is necessary with conventional techniques when the data or thequeries change. The predictive and latent structure query capabilityfurther provides quantification of the uncertainty associated with itspredictions, since the system is built around a fully Bayesianprobability model. For instance, a user may be presented with confidenceindicators or scores of a resulting query, rankings, sorts, and soforth, according to the quality of a prediction rendered.

Described applications built on top of predictive and latent structurequery capability range from predicting heart disease, to understandinghealth care expenditures, to assessing business opportunities andscoring a likelihood to successfully “close” such business opportunities(e.g., to successfully commensurate a sale, contract, etc.).

As noted previously, one of the problems with real-world data is that ittends to be messy with different kinds of data mixed together. Forinstance, structured and unstructured data is commonly blended together,data may be carelessly updated and thus filled with errors, dataelements (e.g., cell or field values) are very often missing resultingin null values or unknown data points, and real-world data is nearlyalways lacking in documentation and is therefore not well understood,even by an organization that has collected and maintained such data, andtherefore the data is not being exploited to its maximum benefit forthat organization. Users of the data may be also be measuring the wrongthing, or may be measuring the same thing in ten different ways.

Such issues arise for various reasons. Perhaps there was never a DBA(Data Base Administrator) for the organization, or the DBA left, and tenyears of sedimentary layers of data has since built up. Or theindividuals responsible for data entry and maintenance simply haveinduced errors through natural human behavior and mistakes. All of theseare very realistic and common problems with “real-world” data found inproduction databases for various organizations, in contrast to pristineand small datasets that may be found in a laboratory or test setting.

The analysis engine described herein which generates the queryableindices in support of the predictive queries must therefore accommodate“real-world” data as it actually exists in the wild. The analysis enginemakes sense of data as it exists in real businesses and does not requirea pristine dataset or data that conforms to idealistic constructs ofwhat data looks like. The analysis engine generates indices which may bequeried for many different questions about many different variables, inreal time. The analysis engine is capable of getting at the hiddenstructures in such data, that is, which variables matter and what arethe segments or groups within the data. At the same time, the analysisengine yields predictive relationships that are trustworthy, that is,through the models utilized by the analysis engine, misleading anderroneous relationships and predictions of a low predictive quality areavoided. Preferably, the analysis engine does not reveal things that arenot true and does not report ghost patterns that may exist in a firstdataset or sample, but do not hold up overall. Such desirablecharacteristics are exceedingly difficult to attain with customizedstatistical analysis and customized predictive modeling, and whollyunheard of in any automated system available to the marketplace today.

When making predictions, it is helpful to additionally let the usersknow whether they can trust the result. That is to say, how confident isthe system is in the result by way of a quantitative measure such as aconfidence indicator, confidence score, confidence interval, etc.Sometimes, it may be necessary for the system to literally respond byindicating: “I do not know” rather than providing a predicted result oflow confidence quality or a result that is below a minimum confidencethreshold set by the user. Accordingly, the system may return a resultthat indicates to the user that an answer is, for example, 1 or between1 and 10 or between negative infinity and positive infinity, each toquantitatively define how confidence the system is in its result.

With probabilities, the system can advise the user that it is, forexample, 90% confident that the answer given is real, accurate, andcorrect, or the system may alternatively return a result indicating thatit simply lacks sufficient data, and thus, there is not enough known bywhich to render a prediction.

According to certain embodiments, the analysis engine utilizes aspecially customized probabilistic model based upon foundational crosscategorization modeling applied to tabular data. Conventionallyavailable cross categorization models provide a good start but maynevertheless be improved. For instance, with conventionally availablecross categorization models it is not possible to run equations.Conversely, the analysis engine implementation described hereinovercomes this deficiency in the conventional arts by enabling suchequation execution. Additionally, conventionally available crosscategorization models relied upon matching data with the chosen model tounderstand hidden structure, much like building a probabilistic index,but requiring the model to be matched to the particular dataset to beanalyzed proved so complex that users of such conventional crosscategorization models required advanced mathematics knowledge andprobability theory understanding merely to select and implement theappropriate model for any given dataset, rendering lay persons whollyincapable of realistically using such models. If an available tool is socomplex that it cannot be utilized by a large segment of the population,then that tool is for all practical purposes, inaccessible to a largesegment of the population, regardless of its existence and availability.

The analysis engine along with the supporting technologies describedherein (e.g., such as the cloud computing interface and PreQL structuredquery language) aims to solve this problem by providing a service whichincludes distributed processing, job scheduling, persistence,check-pointing, and a user-friendly API or front-end interface whichaccepts lay users' questions and queries via the PreQL query structurewhich in turn drastically lowers the learning curve and eliminatesspecially required knowledge necessary to utilize such services. Otherspecialized front end GUIs and interfaces are additionally described tosolve for particular use cases on behalf of users and provide othersimple interfaces to complex problems of probability, thus lowering thecomplexity even further for those particular use cases.

Certain examples of specially implemented use cases include an interfaceto find similar entities so as to enable users to ask questions againsta dataset such as: “What resolved support cases are most like this one?”Or alternatively: “Which previously-won sales opportunities does thispresent opportunity resemble?” Such an interface thus enables users toquery their own dataset for answers that may help them solve a currentproblem, based on similar past solutions or win a current salesopportunity based on past wins that have a similar probabilisticrelationship to the current opportunity profile.

Specific use case implementations additionally assist users inpredicting unknown values. This may be for data that is missing from anotherwise populated dataset, such as null values, or to predict valuesthat are unknowable because they remain in the future. For instance,interfaces may assist the user to predict an answer and associatedconfidence score or indication for questions such as “Will anopportunity be won?” Or “How much will this opportunity be worth ifwon?” Or “How much will I sell this quarter?”

Other use cases may help guide a users' decision making and behavior byasking questions such as “What should I do next to advance thisopportunity?” Or “What additional products should I suggest for thiscustomer?” These tools may be helpful to salespersons directly but alsoto sales managers, and other business individuals affected by the salesprocess.

Notably, an indication of predictive quality can be provided along withpredictions to such questions or simply predictive values provided formissing data. These indications of predictive quality may be referred toas confidence scores, predictive quality indicators, and other names,but generally speaking, they are a reflection of the probability that agiven event or value is likely to occur or likely to be true. There aremany perspectives, but probability may be described as a statement, byan observer, about a degree of belief in an event, past, present, orfuture. Timing does not matter. Thus, probability may be considered as astatement of belief, as follows: “How likely is an event to occur” or“How likely is a value to be true for this dataset?”

Probability is needed because we cannot be 100% certain about what willhappen, or in some instances, what has happened, and as such, theprediction is uncertain. What is uncertainty then? An observer, as notedabove, does not know for sure whether an event will occur,notwithstanding the degree of belief in such an event having occurred,occurring, or to occur in the future.

Probabilities are assigned relative to knowledge or information context.Different observers can have different knowledge, and assign differentprobabilities to same event, or assign different probabilities even whenboth observers have the same knowledge. Probability, as used herein, maybe defined as a number between “0” (zero) and “1” (one), in which 0means the event is certain to not occur on one extreme of a continuumand where 1 means the event is certain to occur on the other extreme ofthe same continuum. Both extremes are interesting because they representa complete absence of uncertainty.

A probability ties belief to one event. A probability distribution tiesbeliefs to every possible event, or at least, every event to beconsidered with a given model. Choosing the outcome space is animportant modeling decision. Summed over all outcomes in space is atotal probability which must be a total of “1,” that is to say, one ofthe outcomes must occur, according to the model. Probabilitydistributions are convenient mathematical forms that help summarize thesystem's beliefs in the various probabilities, but choosing a standarddistribution is a modeling choice in which all models are wrong, butsome are useful.

Consider for example, a Poisson distribution which is a good model whensome event can occur 0 or more times in a span of time. The outcomespace is the number of times the event occurs. The Poisson distributionhas a single parameter, which is the rate, that is, the average numberof times. Its mathematical form has some nice properties, such as,defined for all the non-negative integers sum to 1.

Standard distributions are well known and there are many examplesbesides the Poisson distribution. Each such standard distributionencompasses a certain set of assumptions, such as a particular outcomespace, a particular way of assigning probabilities to outcomes, etc. Ifyou work with them, you'll start to understand why some are nice andsome are frustrating if not outright detrimental to the problem at hand.

But distributions can be even more interesting. The analysis engineutilizes distributions which move beyond the standard distributions withspecially customized modeling thus allowing for a more complex outcomespace and thus further allowing for more complex ways of assigningprobabilities to outcomes. For instance, a mixture distribution combinesmany simpler distributions to form a more complex one. A mixture ofGaussians to model any distribution may be employed, while stillassigning probabilities to outcomes, yielding a more involvedmathematical relationship.

With more complex outcome spaces, a Mondrian process defines adistribution on k-dimensional trees, providing means for dividing up asquare or a cube. The outcome space is defined by all possible trees andresulting divisions look like the famous painting for which the processis named. The resulting outcome space is more structured than what isoffered by the standard distributions. conventional cross categorizationmodels do not use the Mondrian process, but they do use a structuredoutcome space. The analysis engine described herein utilizes theMondrian process in select embodiments. No matter how complex theresulting outcome space is, the analysis engine is capable of alwaysassigning a valid probability to each and every outcome within thedefined outcome space, and each probability assigned represents thedegree of “belief” or the analysis engine's assessment of probabilisticquality according to the models applied as to the likelihood of thegiven outcome, and in which the sum of all probabilities across allpossible outcomes for the space is “1.”

Probabilistic models are utilized because they allow computers to“reason” automatically and systematically, according to the modelsutilized, even in the presence of uncertainty. Probability is thecurrency by which the analysis engine combines varying sources ofinformation to reach the best possible answer in a systematic mannereven when the information is vague, or uncertain, or ambiguous, as isvery often the case with real-world data.

Unbounded categorical data types are additionally used to modelcategorical columns where new values that are not found in the datasetcan show up. For example, most sales opportunities for database serviceswill be replacing one of a handful of common existing systems, such asan Oracle implementation, but a new opportunity might be replacing a newsystem which has not been seen in the data ever before. The priornon-existence of the new value within in the dataset does not mean thatis invalid, and as such, the new value is allowed to be entered for atyped column with a limited set of allowed values (e.g., an enumeratedset) even though it is a previously unseen value. In terms of modeling,the system makes the following inferences: “Where a small number ofvalues in an unbounded categorical data type have been seen heretofore,it is unlikely that a new value will be seen in the future;” and “wherea large number of values in an unbounded categorical data type have beenseen heretofore, it is more likely that a brand new value will be seenin the future.”

FIG. 11 depicts a chart 1101 and graph 1102 of the Bell number series.The Bell numbers define the number of partitions for n labeled objectswhich, as can be seen from the graph 1102 on the right, grow very, veryfast. A handful of objects are exemplified in the chart 1101 on theright. The graph 1102 plots n through 200 resulting in 1e+250 or anumber with 250 zeros. Now consider the massive datasets available in acloud computing multi-tenant database system which may easily result indatasets of interest with thousands of columns and millions of rows.Such datasets will not merely result in the Bell numbers depicted above,but rather, potentially the Bell's “squared,” placing us firmly into thescale of numbers wholly inconceivable by human intellect and experience.

These numbers are so massive that it may be helpful to consider them inthe following context. The hashed line at element 1103 near the bottomof the graph 1102 represents the approximate total quantity of web pagespresently indexed by Google. Google only needs to search through the17th bell number or so. The total space, however, so unimaginablymassive that it simply is not possible to explore it exhaustively.Moreover, because the probability landscape is both vast and ruggedrather than smooth or concave, brute force processing will not work andsimple hill climbing methodologies are not sufficient either.

FIG. 12A depicts an exemplary cross categorization of a small tabulardataset. Here, the exemplary cross categorization consists of view 1 atelement 1201 and view 2 at element 1202. Each of the views 1201 and 1202include both features or characteristics 1204 (depicted as the columns)and entities 1203 (depicted as the rows). Segmenting each of the views1201 and 1202 by whitespace between the entities 1203 (e.g., rows) arecategories 1210, 1211, 1212, 1213, and 1214 within view 1 at element1201 and categories 1215, 1216, 1217, and 1218 within view 2 at element1202. Refer back to FIG. 10A for more examples and explanation aboutviews and categories.

Views 1201 and 1202 pick out a subset of the features 1204 (e.g.,columns) available for a dataset and the respective categories 1210-1218within each view 1201 and 1202 pick out a subset of the entities 1203(e.g. rows). Each column contains a single kind of data so each verticalstrip within a category contains typed data such as numerical,categorical, etc. With such an exemplary cross categorization breakdown,the basic standardized distributions may be utilized more effectively.

In certain embodiments, each collection of points is modeled with asingle simple distribution. Basic distributions that work well for eachdata type may be pre-selected and each selected basic distribution isonly responsible for explaining a small subset of the actual data, forwhich it is particularly useful. Then using the mixture distributiondiscussed above, the basic distributions are combined such that manysimple distributions are utilized to make a more complex one. Thestructure of the cross-categorization is used to chop up the data tableinto a bunch of pieces and each piece is modeled using the simpledistribution selected based on applicable data type(s), yielding a bigmixture distribution of the data.

Referring back to the tabular dataset describing certain mammals asdepicted at FIG. 8, it can now be seen at FIG. 12A what the tabular datalooks like after being subjected to a simplified cross categorizationmodel as shown here with two views 1201 and 1202.

View 1202 on the right includes the habitat and feeding style features1204 (columns) and the entities 1203 (rows) are divided into fourcategories 1215-1218 of land mammals (Persian cat through Zebra), seapredators (dolphin through walrus), baleen whales (blue whale andhumpback whale only), and the outlier amphibious beaver (e.g., both landand water living; we do not suggest that mammal beavers have gills).

View 1201 on the left has another division in which the primates aregrouped together, large mammals are grouped, grazers are grouped, andthen a couple of data oddities at the bottom have been grouped together(bat and seal). Even with a small dataset it is easy to imaginedifferent ways of dividing the data up. But data is ambiguous and thereis no perfect or obviously correct division. For all the groupings thatseemingly fit correctly, certain groupings may seem awkward or poorfitting. The systematic process of applying various models andassumptions makes tradeoffs and compromises, which is why even expertscannot agree on a single approach. Nevertheless, the means describedherein permits use of a variety of available models such that thesetradeoffs and compromises may be exploited systematically by theanalysis engine to an extent and scale that a human expert simplycannot.

For instance, results by the analysis engine are not limited to a singlecross-categorization model or breakdown. Instead, a collection ofcategorization models are utilized and such a collection when usedtogether help to reveal the hidden structure of the data. For instance,if all the resulting categorizations were the same despite the use ofvarying categorization models, then there simply was no ambiguity in thedata. But such a result does not occur with real-world data, despitebeing a theoretical possibility. Conversely, if all the resultingcategorizations are all completely different, then the analysis enginedid not find any structure in the data, which sometimes happens, andwill therefore require some additional post-processing to get at theuncertainty, such as feeding in additional noise. Typically, however,something in between occurs, and some interesting hidden structure isrevealed to the analysis engine from the data through the application ofthe collection of categorization models selected and utilized.

The specially customized cross-categorization implementations representthe processing and logic core of the analysis engine which, due to itsuse and complexity, is intentionally hidden from end users. Rather thanaccessing the analysis engine core directly, users are instead exposedto less complex interfaces via APIs, PreQL, JSON, and other specializedutility GUIs and interfaces which are implemented, for example, by thequery interface depicted at element 180 of FIG. 1. Notwithstanding thislayer of abstraction from analysis engine's core, users neverthelessbenefit from the functionality described without having to possess ahighly specialized understanding of mathematics and probability.

According to certain embodiments, the analysis engine further appliesinference and search to the probability landscape developed, in certaininstances by utilizing Monte Carlo methodologies. For instance, asexemplified by the Bell numbers, the space to be navigated may bemassive. One approach therefore is to simply start somewhere, anywhere,and then compute the probability for the event, outcome, or value atthat location within the available space. Next, another location withinthe space is selected and the probability again computed. Then in aniterative fashion, a determination is made whether to keep the newlocation or instead keep the earlier found location by comparing theprobabilities, and then looping such that a new location is found,probability calculated, compared, and then selected or discarded, and soforth, until a certain amount of time or processing has expired or untila sufficient quality of result is attained (e.g., such as a probabilityor confidence score over a threshold, etc.).

FIG. 12B depicts an exemplary architecture having implemented dataupload, processing, and predictive query API exposure in accordance withdescribed embodiments. In particular, customer organizations 1205A,1205B, and 1205C are depicted, each with a client device 1206A, 1206B,and 1206C capable of interfacing with host organization 1210 via network1225, including sending requests and receiving responses. Within hostorganization 1210 is a request interface 1276 which may optionally beimplemented by web-server 1275. The host organization further includesprocessor(s) 1281, memory 1282, an API interface 1280, analysis engine1285, and a multi-tenant database system 1230. Within the multi-tenantdatabase system 1230 are execution hardware, software, and logic 1220that are shared across multiple tenants of the multi-tenant databasesystem 1230 as well as a predictive database 1250 capable of storingindices generated by the analysis engine to facilitate the return ofpredictive result sets responsive to predictive queries or latentstructure queries executed against the predictive database 1250.

According to one embodiment, the host organization 1210 operates asystem 1211 having at least a processor 1281 and a memory 1282 therein,the system 1211 being enabled to receive tabular datasets as input,process the dataset according to the methodologies described herein,then execute predictive and latent structure query requests receivedagainst indices stored by the predictive database 1250.

In accordance with one embodiment there is a system 1211 that is tooperate within a host organization 1210, in which the system includes atleast: a processor 1281 to execute instructions stored in memory 1282 ofthe system 1211; a request interface 1276 to receive as input a dataset1249 in a tabular form, the dataset 1249 having plurality of rows and aplurality of columns; an analysis engine 1285 to process the dataset1249 and generate indices 1251 representing probabilistic relationshipsbetween the rows and the columns of the dataset 1249; a predictivedatabase 1250 to store the generated indices 1251; the request interface1276 to further receive a request for a predictive and/or latentstructure query 1253 against the indices stored in the predictivedatabase 1250; an Application Programming Interface (API) 1280 to querythe indices stored in the predictive database 1250 for a predictiveresult set 1252 based on the request; and in which the request interface1276 is to return the predictive result set 1252 responsive to therequest received.

In one embodiment, such a system 1211 further includes a web-server 1275to implement the request interface 1276. In such an embodiment, theweb-server 1275 is to receive as input, a plurality of access requestsfrom one or more client devices 1206A-C from among a plurality ofcustomer organizations 1205A-C communicably interfaced with the hostorganization 1210 via a network 1225. According to such an embodiment,the system 1211 further includes a multi-tenant database system 1230with predictive database functionality to implement the predictivedatabase; and further in which each customer organization 1205A-C is anentity selected from the group consisting of: a separate and distinctremote organization, an organizational group within the hostorganization, a business partner of the host organization, or a customerorganization that subscribes to cloud computing services provided by thehost organization.

FIG. 12C is a flow diagram illustrating a method 1221 for implementingdata upload, processing, and predictive query API exposure in accordancewith disclosed embodiments. Method 1221 may be performed by processinglogic that may include hardware (e.g., circuitry, dedicated logic,programmable logic, microcode, etc.), software (e.g., instructions runon a processing device to perform various operations such transmitting,sending, receiving, executing, generating, calculating, storing,exposing, querying, processing, etc., in pursuance of the systems,apparatuses, and methods for implementing data upload, processing, andpredictive query API exposure, as described herein. For example, hostorganization 110 of FIG. 1, machine 400 of FIG. 4, or system 1211 ofFIG. 12B may implement the described methodologies. Some of the blocksand/or operations listed below are optional in accordance with certainembodiments. The numbering of the blocks presented is for the sake ofclarity and is not intended to prescribe an order of operations in whichthe various blocks must occur.

At block 1291, processing logic receives a dataset in a tabular form,the dataset having a plurality of rows and a plurality of columns.

At block 1292, processing logic processes the dataset to generateindices representing probabilistic relationships between the rows andthe columns of the dataset.

At block 1293, processing logic stores the indices in a database.

At block 1294, processing logic exposes an Application ProgrammingInterface (API) to query the indices in the database.

At block 1295, processing logic receives a request for a predictivequery or a latent structure query against the indices in the database.

At block 1296, processing logic queries the database for a result basedon the request via the API.

At block 1297, processing logic returns the result responsive to therequest. For instance, a predictive record set may be returned havingtherein one or more predictions or other elements returned, such as apredictive record set describing group data, similarity data, and/orrelated data.

According to another embodiment of method 1221, processing the datasetincludes learning a joint probability distribution over the dataset toidentify and describe the probabilistic relationships between elementsof the dataset.

According to another embodiment of method 1221, the processing istriggered automatically responsive to receiving the dataset, and inwhich learning the joint probability distribution is controlled by adefault set of configuration parameters.

According to another embodiment of method 1221, learning the jointprobability distribution is controlled by specified configurationparameters, the specified configuration parameters including one or moreof: a maximum period of time for processing the dataset; a maximumnumber of iterations for processing the dataset; a minimum number ofiterations for processing the dataset; a maximum amount of customerresources to be consumed by processing the dataset; a maximum subscriberfee to be expended processing the dataset; a minimum thresholdpredictive quality level to be attained by the processing of thedataset; a minimum improvement to a predictive quality measure requiredfor the processing to continue; and a minimum or maximum number of theindices to be generated by the processing.

According to another embodiment of method 1221, processing the datasetto generate indices includes iteratively learning joint probabilitydistributions over the dataset to generate the indices.

According to another embodiment, the method 1221 further includes:periodically determining a predictive quality measure of the indicesgenerated by the processing of the dataset; and terminating processingof the dataset when the predictive quality measure attains a specifiedthreshold.

According to another embodiment, the method 1221 further includes:receiving a query requesting a prediction from the indices generated byprocessing the dataset; and executing the query against the generatedindices prior to terminating processing of the dataset.

According to another embodiment, the method 1221 further includes:returning a result responsive to the query requesting the prediction;and returning a notification with the result indicating processing ofthe dataset has not yet completed or a notification with the resultindicating the predictive quality measure is below the specifiedthreshold, or both.

According to another embodiment of method 1221, the predictive qualitymeasure is determined by comparing a known result corresponding toobserved and present values within the dataset with a predictive resultobtained by querying the indices generated by the processing of thedataset.

According to another embodiment of method 1221, the predictive qualitymeasure is determined by comparing ground truth data from the data setwith one or more predictive results obtained by querying the indicesgenerated by the processing of the dataset.

According to another embodiment of method 1221, processing the datasetincludes at least one of: learning a Dirichlet Process Mixture Model(DPMM) of the dataset; learning a cross categorization of the dataset;learning an Indian buffet process model of the dataset; and learning amixture model or a mixture of finite mixtures model of the dataset.

According to another embodiment of method 1221, receiving the datasetincludes at least one of the following: receiving the dataset as a tablehaving the columns and rows; receiving the dataset as data stream;receiving a spreadsheet document and extracting the dataset from thespreadsheet document; receiving the dataset as a binary file created bya database; receiving one or more queries to a database and responsivelyreceiving the dataset by executing the one or more queries against thedatabase and capturing a record set returned by the one or more queriesas the dataset; receiving a name of a table in a database and retrievingthe table from the database as the dataset; receiving search parametersfor a specified website and responsively querying the search parametersagainst the specified website and capturing search results as thedataset; and receiving a link and authentication credentials for aremote repository and responsively authenticating with the remoterepository and retrieving the dataset via the link.

According to another embodiment of method 1221, each of the plurality ofrows in the dataset corresponds to an entity; in which each of theplurality of columns corresponds to a characteristic for the entities;and in which a point of intersection between each respective row andeach of the plurality of columns forms a cell to store a value at thepoint of intersection.

According to another embodiment of method 1221, each entity represents aperson, a place, or a thing; and in which each characteristic representsa characteristic, feature, aspect, quantity, range, identifier, mark,trait, or observable fact; in which each cell stores a data typed valueat the point of intersection between each respective row and each of theplurality of columns, the value representing the characteristic for theentity's row that intersects a column corresponding to thecharacteristic; and in which the value of every cell is either null,different, or the same as any other value of any other cell.

According to another embodiment of method 1221, each of the plurality ofcolumns has a specified data type.

According to another embodiment of method 1221, each data typecorresponds to one of: Boolean; a categorical open set; a categoricalclosed set; a set-valued data type defining a collection of values, acollection of identifiers, and/or a collection of strings within adocument; a quantity count; floating point numbers; positive floatingpoint numbers; strings; latitude and longitude pairs; vectors; positiveintegers; a text file; and a data file of a specified file type.

According to another embodiment of method 1221, receiving a dataset in atabular form includes: receiving relational database objects havingmultiple tables with inter-relationships across the multiple tables; andin which processing the dataset includes generating indices from thecolumns and the rows amongst the multiple tables while conforming to theinter-relationships amongst the multiple tables.

For instance, the generative process by which the analysis enginecreates the indices may first divide the features/columns into kinds,and then for each kind identified, the analysis engine next divides theentities/rows into categories. The analysis engine utilizes models thatprovides kinds for which each of the features provide predictiveinformation about other features within the same kind and for which eachcategory contains entities that are similar according to the features inthe respective kind as identified by the model.

PreQL structured queries allow access to the queryable indices generatedby the analysis engine through its modeling via specialized calls,including: “RELATED,” “SIMILAR,” “GROUP,” and “PREDICT.”

According to another embodiment of method 1221, the processing furtherincludes executing Structured Query Language (SQL) operations againsttwo more of the multiple tables to form the dataset; in which the SQLoperations include at least one of an SQL transform operation, an SQLaggregate operation, and an SQL join operation.

According to another embodiment of method 1221, the indices are storedwithin a predictive database system of a host organization; and in whichthe method further includes: receiving a plurality of access requestsfor indices stored within the predictive database system of the hostorganization, each of the access requests originating from one or moreclient devices of a plurality of customer organizations, in which eachcustomer organization is selected from the group consisting of: aseparate and distinct remote organization, an organizational groupwithin the host organization, a business partner of the hostorganization, or a customer organization that subscribes to cloudcomputing services provided by the host organization.

According to another embodiment of method 1221, the predictive databasesystem is operationally integrated with a multi-tenant database systemprovided by the host organization, the multi-tenant database systemhaving elements of hardware and software that are shared by a pluralityof separate and distinct customer organizations, each of the separateand distinct customer organizations being remotely located from the hostorganization having the predictive database system and the multi-tenantdatabase system operating therein.

According to another embodiment of method 1221, receiving a datasetincludes receiving the dataset at a host organization providingon-demand cloud based services that are accessible to remote computingdevices via a public Internet; and in which storing the indices in adatabase includes storing the indices in a predictive database systemoperating at the host organization via operating logic stored in memoryof the predictive database system and executed via one or moreprocessors of the predictive database system.

According to another embodiment of method 1221, storing the indices inthe database includes storing the indices in a predictive database; andin which exposing the API to query the indices includes exposing aPredictive Query Language (PreQL) API.

According to another embodiment of method 1221, receiving the requestfor a predictive query or a latent structure query against the indicesin the database includes receiving a PreQL query specifying at least onecommand selected from the group of PreQL commands including: PREDICT,RELATED, SIMILAR, and GROUP.

According to a particular embodiment there is a non-transitory computerreadable storage medium having instructions stored thereon that, whenexecuted by a processor in a host organization, the instructions causethe host organization to perform operations including: receiving adataset in a tabular form, the dataset having a plurality of rows and aplurality of columns; processing the dataset to generate indicesrepresenting probabilistic relationships between the rows and thecolumns of the dataset; storing the indices in a database; exposing anApplication Programming Interface (API) to query the indices in thedatabase; receiving a request for a predictive query or a latentstructure query against the indices in the database; querying thedatabase for a prediction based on the request via the API; andreturning the prediction responsive to the request.

The non-transitory computer readable storage medium may embody and causeto be performed, any of the methodologies described herein.

In another embodiment, processing of the tabular dataset is triggeredmanually or automatically upon receipt of the tabular dataset as inputat the host organization. When triggered manually, an “UPLOAD” commandmay be issued to pass the tabular dataset to the analysis engine or tospecify a target dataset to the analysis engine for analysis from whichthe predictive indices are generated. In yet another embodiment, an“ANALYZE” command may be issued to instruct the analysis engine toinitiate analysis of a specified dataset. In certain embodiments, theUPLOAD and ANALYZE command terms are used but are hidden from the userand are instead issued by interfaces provided to the user to reducecomplexity of the system for the user.

Functionality of the analysis engine which generates the indices fromthe tabular datasets is computationally intensive and is thus, is wellsuited for a distributed computing structure provided by a cloud basedmulti-tenant database system architecture.

According to the described embodiments, the resulting database appearsto its users much like a traditional database. But instead of selectingcolumns from existing rows, users may issue predictive query requestsvia a structured query language. Such a structured language, rather thanSQL may be referred to as Predictive Query Language (“PreQL”). PreQL isnot to be confused with PQL which is short for the “Program QueryLanguage.”

PreQL is thus used to issue queries against the database to predictvalues, events, or outcomes according to models applied to the datasetat hand by the analysis engine and its corresponding functionality. Sucha PreQL query offers the same flexibility as SQL-style queries. Whenexploring structure, users may issue PreQL queries seeking notions ofsimilarity that are hidden or latent in the overall data withoutadvanced knowledge of what those similarities may be. Users may issuepredictive queries seeking notions of relatedness amongst the columnswithout having to know those relations before hand. Users may issuepredictive queries seeking notions of groupings amongst entities withinthe dataset without having to know or define such groupings or rules forsuch groupings before hand. And when used within a multi-tenant databasesystem against a massive cloud based database and its dataset, suchfeatures are potentially transformative in the computing arts.

FIG. 12D depicts an exemplary architecture having implemented predictivequery interface as a cloud service in accordance with describedembodiments. In particular, customer organizations 1205A, 1205B, and1205C are depicted, each with a client device 1206A, 1206B, and 1206Ccapable of interfacing with host organization 1210 via public Internet1228, including sending queries (e.g., input 1257) and receivingresponses (e.g., predictive record set 1258). Within host organization1210 is a request interface 1276 which may optionally be implemented byweb-server 1275. The host organization further includes processor(s)1281, memory 1282, a query interface 1280, analysis engine 1285, and amulti-tenant database system 1230. Within the multi-tenant databasesystem 1230 are execution hardware, software, and logic 1220 that areshared across multiple tenants of the multi-tenant database system 1230,authenticator 1298, one or more application servers 1265, as well as apredictive database 1250 capable of storing indices generated by theanalysis engine 1285 to facilitate the return of the predictive recordset (1258) responsive to predictive queries and/or latent structurequeries (e.g., requested via input 1257) executed against the predictivedatabase 1250.

According to one embodiment, the host organization 1210 operates asystem 1231, in which the system 1231 includes at least: a processor1281 to execute instructions stored in memory 1282 of the system 1231; arequest interface 1276 exposed to client devices 1206A-C that operateremotely from the host organization 1210, in which the request interface1276 is accessible by the client devices 1206A-C via a public Internet1228; and a predictive database 1250 to execute as an on-demand cloudbased service for one or more subscribers, such as those operating theclient devices 1206A-C or are otherwise affiliated with the variouscustomer organizations 1205A-C to which such services are provided.According to such an embodiment, such a system 1231 further includes anauthenticator 1298 to verify that client devices 1206A-C are associatedwith a subscriber and to further verify authentication credentials forthe respective subscriber; in which the request interface is to receiveas input, a request from the subscriber; the system 1231 furtherincluding one or more application servers 1265 to execute a query (e.g.,provided as input 1257 via the public Internet 1228) against indices ofthe predictive database 1250 generated from a dataset of columns androws on behalf of the subscriber, in which the indices representprobabilistic relationships between the rows and the columns of thedataset; and in which the request interface 1276 of the system 1231 isto further return a predictive record set 1258 to the subscriberresponsive to the request.

According to another embodiment, such a system 1231 further includes aweb-server 1275 to implement the request interface, in which theweb-server 1275 is to receive as input 1257, a plurality of accessrequests from one or more client devices 12066A-C from among a pluralityof customer organizations 1205A-C communicably interfaced with the hostorganization via a network traversing at least a portion of the publicInternet 1228, in which each customer organization 105A-C is an entityselected from the group consisting of: a separate and distinct remoteorganization, an organizational group within the host organization, abusiness partner of the host organization, or a customer organizationthat subscribes to cloud computing services provided by the hostorganization 1210. According to such an embodiment, a multi-tenantdatabase system 1230 at the host organization 1210 with predictivedatabase functionality is to implement the predictive database.

According to another embodiment, system 1231 includes an analysis engine1285 to process the dataset and to generate the indices representingprobabilistic relationships between the rows and the columns of thedataset, and further in which the predictive database 1250 is to storethe generated indices. According to another embodiment of the system1231, a Predictive Query Language Application Programming Interface(PreQL API 1299) is exposed to the subscribers at the request interface1276, in which the PreQL API 1299 accepts PreQL queries (e.g., as input1257) having at least at least one command selected from the group ofPreQL commands including: PREDICT, RELATED, SIMILAR, and GROUP,subsequent to which the PreQL API 1299 executes the PreQL queriesagainst the predictive database and returns a predictive record set1258.

In such a way, use of the PreQL structure queries permits programmaticqueries into the indices generated and stored within the predictivedatabase in a manner similar to a programmer making SQL queries into arelational database. Rather than a “SELECT” command term, a variety ofpredictive PreQL based command terms are instead utilized, such as the“PREDICT” or “SIMILAR” or “RELATED” or “GROUP” statements. For instance,an exemplary PreQL statement may read as follows: “PREDICT IS_WON,DOLLAR_AMOUNT FROM OPPORTUNITY WHERE STAGE=‘QUOTE’.” So in this example,“QUOTE” is the fixed column, “FROM” is the dataset from which anopportunity is to be predicted, the “PREDICT” command term is the callinto the appropriate function, “IS_WON” is the value to be predicted,that is to say, the functionality is to predict whether a givenopportunity is likely or unlikely to be won where the “IS_WON” may havecompleted data for some rows but be missing for other rows due to, forexample, pending or speculative opportunities, etc. “DOLLAR_AMOUNT” isthe fixed value.

In certain embodiments, the above query is implemented via a specializedGUI interface which accepts inputs from a user via the GUI interface andconstructs, calls, and returns data via the PREDICT functionality onbehalf of the user without requiring the user actually write or even beaware of the underlying PreQL structure query made to the analysisengine's core.

Another exemplary PreQL statement may read as follows: SELECT ID; FROMOpportunity WHERE SIMILAR/Stage/001>0.8 ORDER BY SIMILAR/Stage LIMIT100. In this example, a particular ID is being pulled from the“Opportunity” table in the database and then the SIMILAR command term isused to find identify the entities or rows similar to the ID specified,so long as they have a confidence quality indicator equal to or greaterthan “0.8,” and finally the output is ordered by stage and permitted toyield output of up to 100 total records. This particular exampleutilizes a mixture of both SQL and PreQL within the query (e.g., the“SELECT” command term is a SQL command and the “SIMILAR” command term isspecific to PreQL).

FIG. 12E is a flow diagram illustrating a method for implementingpredictive query interface as a cloud service in accordance withdisclosed embodiments. Method 1222 may be performed by processing logicthat may include hardware (e.g., circuitry, dedicated logic,programmable logic, microcode, etc.), software (e.g., instructions runon a processing device to perform various operations such transmitting,sending, receiving, executing, generating, calculating, storing,exposing, authenticating, querying, processing, returning, etc., inpursuance of the systems, apparatuses, and methods for implementingpredictive query interface as a cloud service, as described herein. Forexample, host organization 110 of FIG. 1, machine 400 of FIG. 4, orsystem 1231 of FIG. 12D may implement the described methodologies. Someof the blocks and/or operations listed below are optional in accordancewith certain embodiments. The numbering of the blocks presented is forthe sake of clarity and is not intended to prescribe an order ofoperations in which the various blocks must occur.

At block 1270, processing logic exposes an interface to client devicesoperating remotely from the host organization, in which the interface isaccessible by the client devices via a public Internet.

At block 1271, processing logic executes a predictive database at thehost organization as an on-demand cloud based service for one or moresubscribers.

At block 1272, processing logic authenticates one of the client devicesby verifying the client device is associated with one of the subscribersand based further on authentication credentials for the respectivesubscriber.

At block 1273, processing logic receives a request from theauthenticated subscriber via the interface.

At block 1274, processing logic executes a predictive query or a latentstructure query against indices of the predictive database generatedfrom a dataset of columns and rows on behalf of the authenticatedsubscriber, the indices representing probabilistic relationships betweenthe rows and the columns of the dataset.

At block 1279, processing logic returns a predictive record set to theauthenticated subscriber responsive to the request.

According to another embodiment of method 1222, executing the queryincludes executing a Predictive Query Language (PreQL) query against thepredictive database.

According to another embodiment of method 1222, executing the PreQLquery includes querying the predictive database by specifying at leastone command selected from the group of PreQL commands including:PREDICT, RELATED, SIMILAR, and GROUP.

According to another embodiment of method 1222, receiving the requestincludes receiving the PreQL query from the authenticated subscriber viaa Predictive Query Language (PreQL) API exposed directly to theauthenticated subscriber via the interface.

According to another embodiment of method 1222, receiving the requestincludes: presenting a web form to the authenticated subscriber;receiving inputs from the authenticated subscriber via the web form;generating a PreQL query on behalf of the authenticated subscriber basedon the inputs; querying the predictive database via a Predictive QueryLanguage (PreQL) API by specifying at least one command selected fromthe group of PreQL commands including: PREDICT, RELATED, SIMILAR, andGROUP.

According to another embodiment of method 1222, the authenticatedsubscriber accesses the on-demand cloud based service via a web-browserprovided by a third party different than the host organization; and inwhich the authenticated subscriber submits the request to the hostorganization and receives the predictive record set from the hostorganization without installing any software from the host organizationon the client device.

According to another embodiment, method 1222 further includes: receivingthe dataset from the authenticated subscriber prior to receiving therequest from the authenticated subscriber; and processing the dataset onbehalf of the authenticated subscriber to generate the indices, each ofthe indices representing probabilistic relationships between the rowsand the columns of the dataset.

According to another embodiment of method 1222, receiving the datasetincludes at least one of: receiving the dataset as a table having thecolumns and rows; receiving the dataset as data stream; receiving aspreadsheet document and extracting the dataset from the spreadsheetdocument; receiving the dataset as a binary file created by a database;receiving one or more queries to a database and responsively receivingthe dataset by executing the one or more queries against the databaseand capturing a record set returned by the one or more queries as thedataset; receiving a name of a table in a database and retrieving thetable from the database as the dataset; receiving search parameters fora specified website and responsively querying the search parametersagainst the specified website and capturing search results as thedataset; and receiving a link and authentication credentials for aremote repository and responsively authenticating with the remoterepository and retrieving the dataset via the link.

According to another embodiment of method 1222, processing the dataseton behalf of the authenticated subscriber includes learning a jointprobability distribution over the dataset to identify and describe theprobabilistic relationships between elements of the dataset.

According to another embodiment of method 1222, the processing istriggered automatically responsive to receiving the dataset, and inwhich learning the joint probability distribution is controlled byspecified configuration parameters, the specified configurationparameters including one or more of: a maximum period of time forprocessing the dataset; a maximum number of iterations for processingthe dataset; a minimum number of iterations for processing the dataset;a maximum amount of customer resources to be consumed by processing thedataset; a maximum subscriber fee to be expended processing the dataset;a minimum threshold confidence quality level to be attained by theprocessing of the dataset; a minimum improvement to a confidence qualitymeasure required for the processing to continue; and a minimum ormaximum number of the indices to be generated by the processing.

According to another embodiment of method 1222, processing the datasetincludes iteratively learning joint probability distributions over thedataset to generate the indices; and in which the method furtherincludes: periodically determining a predictive quality measure of theindices generated by the processing of the dataset; and terminatingprocessing of the dataset when the confidence quality measure attains aspecified threshold.

According to another embodiment, method 1222 further includes: returninga notification with the predictive record set indicating processing ofthe stored dataset has not yet completed or a notification with thepredictive record set indicating the confidence quality measure is belowthe specified threshold, or both.

According to another embodiment of method 1222, the confidence qualitymeasure is determined by comparing a known result corresponding to knownand non-null values within the dataset with a predictive record setobtained by querying the indices generated by the processing of thedataset.

According to another embodiment of method 1222, the host organizationincludes a plurality of application servers; in which the processingfurther includes distributing the generation of the indices and storingof the indices amongst multiple of the application servers; in whichexecuting the query against indices of the predictive database includesquerying multiple of the generated indices among multiple of theapplication servers to which the indices were distributed and stored;and aggregating results returned by the querying of the multiple of thegenerated indices.

According to another embodiment of method 1222, querying the generatedindices in parallel yields different results from different versions ofthe indices at the multiple of the application servers to which theindices were distributed and stored.

According to another embodiment, method 1222 further includes:aggregating the different results; and returning one predictive recordset responsive to an executed latent structure query or one predictionresponsive to an executed predictive query.

According to another embodiment of method 1222, a greater quantity ofthe generated indices corresponds to an improved prediction accuracy;and in which a greater quantity of the application servers to which theindices are distributed and stored corresponds to an improved responsetime for executing the query.

According to another embodiment, method 1222 further includes: receivinga specified data source from the authenticated subscriber; andperiodically updating the indices based on the specified data source.

According to another embodiment of method 1222, periodically updatingthe indices includes one of: initiating a polling mechanism to check forchanges at the specified data source and retrieving the changes whendetected for use in updating the indices; receiving push notificationsfrom the specified data source indicating changes at the specified datasource have occurred and accepting the changes for use in updating theindices; and in which the updating of the indices occurs withoutrequiring an active authenticated session for the subscriber.

According to another embodiment, method 1222 further includes: executingStructured Query Language (SQL) operations against two or more tableswithin the host organization which are accessible to the authenticatedsubscriber, in which the SQL operations include at least one of an SQLtransform operation, an SQL aggregate operation, and an SQL joinoperation; capturing the output of the SQL operations as the dataset ofrows and columns; and processing the dataset to generate the indicesrepresenting the probabilistic relationships between the rows and thecolumns of the dataset.

According to another embodiment of method 1222, the authenticatedsubscriber specifies the two or more tables as input and in which thehost organization generates a query to perform the SQL operations andautomatically initiates processing against the dataset on behalf of theauthenticated subscriber.

According to another embodiment, method 1222 further includes:generating the indices representing probabilistic relationships betweenthe rows and the columns of the dataset by learning at least one of:learning a Dirichlet Process Mixture Model (DPMM) of the dataset;learning a cross categorization of the dataset; learning an Indianbuffet process model of the dataset; and learning a mixture model or amixture of finite mixtures model of the dataset.

According to another embodiment of method 1222, each of the plurality ofrows in the dataset corresponds to an entity; in which each of theplurality of columns corresponds to a characteristic for the entities;and in which a point of intersection between each respective row andeach of the plurality of columns forms a cell to store a value at thepoint of intersection.

According to another embodiment of method 1222, each entity represents aperson, a place, or a thing; and in which each characteristic representsa characteristic, feature, aspect, quantity, range, identifier, mark,trait, or observable fact; in which each cell stores a data typed valueat the point of intersection between each respective row and each of theplurality of columns, the value representing the characteristic for theentity's row that intersects a column corresponding to thecharacteristic; and in which the may be pre-selected value of every cellis either null, different, or the same as any other value of any othercell.

According to another embodiment, there is a non-transitory computerreadable storage medium having instructions stored thereon that, whenexecuted by a processor in a host organization, the instructions causethe host organization to perform operations including: exposing aninterface to client devices operating remotely from the hostorganization, in which the interface is accessible by the client devicesvia a public Internet; executing a predictive database at the hostorganization as an on-demand cloud based service for one or moresubscribers; authenticating one of the client devices by verifying theclient device is associated with one of the subscribers and basedfurther on authentication credentials for the respective subscriber;receiving a request from the authenticated subscriber via the interface;executing a query against indices of the predictive database generatedfrom a dataset of columns and rows on behalf of the authenticatedsubscriber, the indices representing probabilistic relationships betweenthe rows and the columns of the dataset; and returning a predictiverecord set to the authenticated subscriber responsive to the request.

FIG. 13A illustrates usage of the RELATED command term in accordancewith the described embodiments. Specialized queries are made feasibleonce the analysis engine generates the indices from the tabulardataset(s) provided as described above. For instance, users can ask thepredictive database: “For a given column, what are the other columnsthat are predictively related to it?” In the language of the queryableindices, this translates to: “How often does each other column appearwithin the same view” as is depicted at element 1302. In terms of thecross-categorizations, the analysis engine tabulates how often each ofthe other columns appears in the same view as the input column, thusrevealing what matters and what does not matter. All that a user needsto provide as input is a column ID 1301 with the use of the RELATEDcommand term.

Additional predictive functionality as provided by the RELATED commandterm enables users to query for columns that are related to a specifiedcolumn according to the probabilistic models. For example, given a tablewith columns or variables in it, the analysis engine divides the columnsor variables into groups and because of the distributions there aremultiple ways in which to divide up the columns or variables. Takeheight for example. Giving the height column to an API call for theRELATED command term, a user can query: “How confident can I be aboutthe probability of the relationship existing in all the other columnswith the height column specified?” The RELATED command term will thenreturn for the specified height column, a confidence indicator for everyother column in the dataset which was not specified. So for example, theRELATED functionality may return for its confidence indicator to theheight column, “Weight=1.0,” meaning that the analysis engine, accordingto the dataset, is extremely confident that there is a relationshipbetween weight and height.

Such a result is somewhat intuitive and expected, but other results maybe less intuitive and thus provide interesting results for explorationand additional investigation. Continuing with the “height” example forthe specified column to a RELATED command term API call, the analysesengine may return “Age=0.8” meaning that the analyses engine hasdetermined Age to be highly correlated from a probabilistic standpointfor the dataset analyzed, but not perfectly certain as was the case withweight. The lesser confidence indicator score may be due to, forinstance, noisy data which precludes an absolute positive result.

Perhaps also returned for the specified “height” column is “haircolor=0.1” meaning there is realistically no correlation whatsoeverbetween a person's height and their hair color, according to the datasetutilized. Thus, the RELATED functionality permits a user to query forwhat matters for a given column, such as the height column, and thefunctionality returns all the columns with a scoring of how related thecolumns are to the specified column, based on their probability. Whileit may be intuitive for humans to understand that height and weight arerelated, the analysis engine generates such a result systematicallywithout human input and more importantly, can be applied to datasets forwhich such relationships are not intuitive or easily understood by ahuman viewing the data. This is especially true with larger datasets inwhich relationships are sure to exist, but for which the relationshipsare not defined by the data structure itself. The analysis engine learnsthe underlying latent structure and latent relationships which in turnhelp to reveal hidden structure to even lay users wishing to exploretheir data in ways that historically were simply not feasible.

FIG. 13B depicts an exemplary architecture in accordance with describedembodiments. In particular, customer organizations 1305A, 1305B, and1305C are depicted, each with a client device 1306A, 1306B, and 1306Ccapable of interfacing with host organization 1310 via network 1325,including sending queries and receiving responses. Within hostorganization 1310 is a request interface 1376 which may optionally beimplemented by web-server 1375. The host organization further includesprocessor(s) 1381, memory 1382, a query interface 1380, analysis engine1385, and a multi-tenant database system 1330. Within the multi-tenantdatabase system 1330 are execution hardware, software, and logic 1320that are shared across multiple tenants of the multi-tenant databasesystem 1330, authenticator 1398, and a predictive database 1350 capableof storing indices generated by the analysis engine to facilitate thereturn of predictive record sets responsive to queries executed againstthe predictive database 1350.

According to one embodiment, the host organization 1310 operates asystem 1311 having at least a processor 1381 and a memory 1382 therein,the system 1311 being enabled to generate indices from a dataset ofcolumns and rows via the analysis engine 1385, in which the indicesrepresent probabilistic relationships between the rows and the columnsof the dataset. Such a system 1311 further includes the predictivedatabase 1350 to store the indices; a request interface 1376 to exposethe predictive database, for example, to users or to the client devices1306A-C, in which the request interface 1376 is to receive a query 1353for the predictive database specifying a RELATED command term and aspecified column as a parameter for the RELATED command term; a queryinterface 1380 to query the predictive database 1350 using the RELATEDcommand term and pass the specified column to generate a predictiverecord set 1354; and in which the request interface 1376 is to furtherreturn the predictive record set 1354 responsive to the query. In suchan embodiment, the predictive record set 1354 includes a plurality ofelements 1399 therein, each of the returned elements including a columnidentifier and a confidence indicator for the specified column passedwith the RELATED command term. In such an embodiment, the confidenceindicator indicates whether a latent relationship exists between thespecified column passed with the RELATED command and the columnidentifier returned for the respective element 1399.

According to one embodiment, the predictive database 1350 is to executeas an on-demand cloud based service at the host organization 1310 forone or more subscribers. In such an embodiment, the system furtherincludes an authenticator 1398 to verify that client devices 1306A-C areassociated with a subscriber and to further verify authenticationcredentials for the respective subscriber.

According to one embodiment, the request interface 1376 exposes aPredictive Query Language Application Programming Interface (PreQL API)directly to authenticated users, in which the PreQL API is accessible tothe authenticated users via a public Internet. For example, network 1325may operate to link the host organization 1310 with subscribers over thepublic Internet.

According to one embodiment, such a system 1311 includes a web-server1375 to implement the request interface 1376 in which the web-server1375 is to receive as input, a plurality of access requests from one ormore client devices 1306A-C from among a plurality of customerorganizations 1305A-C communicably interfaced with the host organizationvia a network. In such an embodiment, a multi-tenant database system1330 with predictive database functionality implements the predictivedatabase 1350.

FIG. 13C is a flow diagram illustrating a method 1321 in accordance withdisclosed embodiments. Method 1321 may be performed by processing logicthat may include hardware (e.g., circuitry, dedicated logic,programmable logic, microcode, etc.), software (e.g., instructions runon a processing device to perform various operations such transmitting,sending, receiving, executing, generating, calculating, storing,exposing, querying, processing, etc., in pursuance of the systems,apparatuses, and methods for implementing a RELATED command with apredictive query interface, as described herein. For example, hostorganization 110 of FIG. 1, machine 400 of FIG. 4, or system 1311 ofFIG. 13B may implement the described methodologies. Some of the blocksand/or operations listed below are optional in accordance with certainembodiments. The numbering of the blocks presented is for the sake ofclarity and is not intended to prescribe an order of operations in whichthe various blocks must occur.

At block 1391, processing logic generates indices from a dataset ofcolumns and rows, the indices representing probabilistic relationshipsbetween the rows and the columns of the dataset.

At block 1392, processing logic stores the indices within a database ofthe host organization.

At block 1393, processing logic exposes the database of the hostorganization via a request interface.

At block 1394, processing logic receives, at the request interface, aquery for the database specifying a RELATED command term and a specifiedcolumn as a parameter for the RELATED command term.

At block 1395, processing logic queries the database using the RELATEDcommand term and passes the specified column to generate a predictiverecord set.

At block 1396, processing logic returns the predictive record setresponsive to the query, the predictive record set having a plurality ofelements therein. In such an embodiment, each of the returned elementsinclude a column identifier and a confidence indicator for the specifiedcolumn passed with the RELATED command term, in which the confidenceindicator indicates whether a latent relationship exists between thespecified column passed with the RELATED command and the columnidentifier returned for the respective element.

According to another embodiment, method 1321 further includes: passing aminimum confidence threshold with the RELATED command term. In such anembodiment, returning the predictive record set includes returning onlythe elements of the predictive record set having a confidence indicatorin excess of the minimum confidence threshold.

According to another embodiment, method 1321 further includes: passing arecord set limit with the RELATED command term to restrict a quantity ofelements returned with the predictive record set.

According to another embodiment of method 1321, the elements of thepredictive record set are returned ordered by descending order accordingto a confidence indicator for each of the elements of the predictiverecord set or are returned ordered by ascending order according to theconfidence indicator for each of the elements of the predictive recordset.

According to another embodiment of method 1321, the predictively relatedcolumns included with each element returned within the predictive recordset are based further on a fraction of times the predictively relatedcolumns occur in a same column grouping as the specified column passedwith the RELATED command term.

According to another embodiment of method 1321, the predictive recordset having a plurality of elements therein includes each of the returnedelements including all of the columns and a corresponding predictedvalue for every one of the columns; and in which the method furtherincludes returning a confidence indicator for each of the correspondingpredicted values ranging from 0 indicating a lowest possible level ofconfidence in the respective predicted value to 1 indicating a highestpossible level of confidence in the respective predicted value.

According to another embodiment, method 1321 further includes:identifying one or more of the predictively related columns from thepredictive record set generated responsive to the querying the databaseusing the RELATED command term based on a minimum threshold for thepredictively related columns; and inputting the identified one or moreof the predictively related columns into a second query specifying aPREDICT command term or a GROUP command term to restrict a secondpredictive record set returned from the second query.

According to another embodiment of method 1321, querying the databaseusing the RELATED command term includes the database estimating mutualinformation between the specified column passed with the RELATED commandterm and the column identifier returned for the respective element ofthe predictive record set.

According to another embodiment of method 1321, exposing the database ofthe host organization includes exposing a Predictive Query LanguageApplication Programming Interface (PreQL API) directly to authenticatedusers, in which the PreQL API is accessible to the authenticated usersvia a public Internet.

According to another embodiment of method 1321, querying the databaseusing the RELATED command term includes passing a PreQL query to thedatabase, the PreQL query having a query syntax of: the RELATED commandterm as a required term; an optional FROM term specifying one or moretables, datasets, data sources, and/or indices to be queried when theoptional FROM term is specified and in which a default value is used forthe one or more tables, datasets, data sources, and/or indices to bequeried when the optional FROM term is not specified; and a TARGET termspecifying the column to be passed with the RELATED command term. Forexample, if the FROM term goes unspecified, then the system maydetermine a source based on context of the user. For instance, the usermay be associated with a particular organization having only one datasource, or having a primary data source, or the last assessed or mostfrequently accesses data source may be assumed, or a default may bepre-configured as a user preference, and so forth.

According to another embodiment of method 1321, the query syntax for thePreQL query further provides one or more of: an optional CONFIDENCE termthat, when provided, specifies the minimum acceptable confidenceindicator to be returned with the predictive record set; an optionalCOUNT term that, when provided, specifies a maximum quantity of elementsto be returned within the predictive record set; and an optional ORDERBY term that, when provided, specifies whether the elements of thepredictive record are to be returned in ascending or descending orderaccording to a confidence indicator for each of the elements returnedwith the predictive record set.

According to another embodiment of method 1321, querying the databaseusing the RELATED command term includes passing a JavaScript ObjectNotation (JSON) structured query to the database, the JSON structuredquery having a query syntax of: the RELATED command term as a requiredterm; an optional one or more tables, datasets, data sources, and/orindices to be queried or a default value for the one or more tables,datasets, data sources, and/or indices to be queried when not specified;the column to be passed with the RELATED command term; an optionalspecification of a minimum acceptable confidence to be returned with thepredictive record set according to a confidence indicator; an optionalspecification of a maximum quantity of elements to be returned withinthe predictive record set; and an optional specification of whether theelements of the predictive record are to be returned in ascending ordescending order according to a confidence indicator for each of theelements returned with the predictive record set.

According to another embodiment of method 1321, exposing the database ofthe host organization includes exposing a web form directly toauthenticated users, in which the web form is accessible to theauthenticated users via a public Internet; in which the hostorganization generates a latent structure query for submission to thedatabase based on input from the web form; and in which querying thedatabase using the RELATED command term includes querying the databaseusing the latent structure query via a Predictive Query LanguageApplication Programming Interface (PreQL API) within the hostorganization, the PreQL API being indirectly exposed to authenticatedusers through the web form.

According to another embodiment of method 1321, querying the databaseusing the RELATED command term includes executing a Predictive QueryLanguage (PreQL) structured query against the database for the RELATEDcommand term; and in which the method further includes executing one ormore additional PreQL structured queries against the database, each ofthe one or more additional PreQL structured queries specifying at leastone command selected from the group of PreQL commands including:PREDICT, RELATED, SIMILAR, and GROUP.

According to another embodiment, method 1321 further includes: receivingthe dataset from an authenticated subscriber and subsequently receivingthe query for the database from the authenticated subscriber; andprocessing the dataset on behalf of the authenticated subscriber togenerate the indices.

According to another embodiment of method 1321, each of the plurality ofrows in the dataset corresponds to an entity; in which each of theplurality of columns corresponds to a characteristic for the entities;and in which a point of intersection between each respective row andeach of the plurality of columns forms a cell to store a value at thepoint of intersection.

According to a particular embodiment, there is a non-transitory computerreadable storage medium having instructions stored thereon that, whenexecuted by a processor in a host organization, the instructions causethe host organization to perform operations including: generatingindices from a dataset of columns and rows, the indices representingprobabilistic relationships between the rows and the columns of thedataset; storing the indices within a database of the host organization;exposing the database of the host organization via a request interface;receiving, at the request interface, a query for the database specifyinga RELATED command term and a specified column as a parameter for theRELATED command term; querying the database using the RELATED commandterm and passing the specified column to generate a predictive recordset; and returning the predictive record set responsive to the query,the predictive record set having a plurality of elements therein, eachof the returned elements including a column identifier and a confidenceindicator for the specified column passed with the RELATED command term,in which the confidence indicator indicates whether a latentrelationship exists between the specified column passed with the RELATEDcommand and the column identifier returned for the respective element.

FIG. 14A illustrates usage of the GROUP command term in accordance withthe described embodiments. Using the GROUP command term users can ask:“What rows go together.” Such a feature can be conceptualized asclustering, except that there's more than one way to cluster thedataset. Consider the mammals example used above in which a column ID1401 is provided as input. Responsive to such a query, a predictivedataset will be returned as output 1402 having groups in the context ofthe column provided. More particularly, the output 1402 will indicatewhich rows most often appear together as a group in the same categoriesin the view that contains the input column.

Sometimes rows tend to group up on noisy elements in a dataset when theanalysis engine applies its modeling to generate the indices; yet thesenoisy elements sometimes result in groupings that are not actuallyimportant. Using the GROUP command term functionality a user knows thateach column will appear in exactly one of the groups as a view and sothe analysis engine permits a user specified column to identify theparticular “view” that will be utilized. The GROUP functionalitytherefore implements a row centric operation like the SIMILARfunctionality, but in contrast to an API call for SIMILAR where the userspecifies the row and responsively receives back a list of other rowsand corresponding scores based on their probabilities of being similar,the GROUP functionality requires no row to be specified or fixed by theuser. Instead, only a column is required to be provided by the user whenmaking a call to specifying GROUP command term.

Calling the GROUP functionality with a specified or fixed column causesthe functionality to return the groupings of the ROWS that seem to berelated or correlated in some way based on analysis engine's modeling.

FIG. 14B depicts an exemplary architecture in accordance with describedembodiments. In particular, customer organizations 1405A, 1405B, and1405C are depicted, each with a client device 1406A, 1406B, and 1406Ccapable of interfacing with host organization 1410 via network 1425,including sending queries and receiving responses. Within hostorganization 1410 is a request interface 1476 which may optionally beimplemented by web-server 1475. The host organization further includesprocessor(s) 1481, memory 1482, a query interface 1480, analysis engine1485, and a multi-tenant database system 1430. Within the multi-tenantdatabase system 1430 are execution hardware, software, and logic 1420that are shared across multiple tenants of the multi-tenant databasesystem 1430, authenticator 1498, and a predictive database 1450 capableof storing indices generated by the analysis engine to facilitate thereturn of predictive record sets responsive to queries executed againstthe predictive database 1450 by a query interface.

According to one embodiment, the host organization 1410 operates asystem 1411 having at least a processor 1481 and a memory 1482 therein,in which the system 1411 includes an analysis engine 1485 to generateindices from a dataset of columns and rows, in which the indicesrepresent probabilistic relationships between the rows and the columnsof the dataset. Such a system 1411 further includes the predictivedatabase 1450 to store the indices; a request interface 1476 to exposethe predictive database, for example, to users or to the client devices1406A-C, in which the request interface 1476 is to receive a query 1453for the predictive database specifying a GROUP command term and aspecified column as a parameter for the GROUP command term; a queryinterface 1480 to query the predictive database 1450 using the GROUPcommand term and passing the specified column to generate a predictiverecord set 1454; and in which the request interface 1476 is to furtherreturn the predictive record set 1454 responsive to the query 1453, inwhich the predictive record set includes a plurality of groups 1499specified therein, each of the returned groups 1499 of the predictiverecord set including a group of one or more rows of the dataset. Forexample, in the predictive record set 1454 depicted there are fourgroups returned, Group_A 1456; Group_B 1457; Group_C 1458; and Group_D1459, each of which includes a set of {rows}.

FIG. 14C is a flow diagram illustrating a method in accordance withdisclosed embodiments.

Method 1421 may be performed by processing logic that may includehardware (e.g., circuitry, dedicated logic, programmable logic,microcode, etc.), software (e.g., instructions run on a processingdevice to perform various operations such transmitting, sending,receiving, executing, generating, calculating, storing, exposing,querying, processing, etc., in pursuance of the systems, apparatuses,and methods for implementing a GROUP command with a predictive queryinterface, as described herein. For example, host organization 110 ofFIG. 1, machine 400 of FIG. 4, or system 1411 of FIG. 14B may implementthe described methodologies. Some of the blocks and/or operations listedbelow are optional in accordance with certain embodiments. The numberingof the blocks presented is for the sake of clarity and is not intendedto prescribe an order of operations in which the various blocks mustoccur.

At block 1491, processing logic generates indices from a dataset ofcolumns and rows, the indices representing probabilistic relationshipsbetween the rows and the columns of the dataset.

At block 1492, processing logic stores the indices within a database ofthe host organization.

At block 1493, processing logic exposes the database of the hostorganization via a request interface.

At block 1494, processing logic receives, at the request interface, aquery for the database specifying a GROUP command term and a specifiedcolumn as a parameter for the GROUP command term.

At block 1495, processing logic queries the database using the GROUPcommand term and passes the specified column to generate a predictiverecord set.

At block 1496, processing logic returns the predictive record setresponsive to the query, the predictive record set having a plurality ofgroups specified therein, each of the returned groups of the predictiverecord set including a group of one or more rows of the dataset.

According to another embodiment of method 1421, all of the rows of thedataset are partitioned by assigning every row of the dataset to exactlyone of the plurality of groups returned with the predictive record setwithout overlap of any single row being assigned to more than one of theplurality of groups.

According to another embodiment of method 1421, the rows of the datasetare segmented by assigning rows of the dataset to at most one of theplurality of groups without overlap of any single row being assigned tomore than one of the plurality of groups; in which the segmentationresults in one or more rows of the dataset remaining unassigned to anyof the plurality of groups due to a confidence indicator for thecorresponding one or more rows remaining unassigned falling below aminimum threshold.

According to another embodiment of method 1421, a confidence indicatorreturned with each of the one or more rows specified within each of theplurality of groups returned with the predictive record set ranges froma minimum of 0 indicating a lowest possible confidence in the predictionthat the respective row belongs to the group specified to a maximum of 1indicating a highest possible confidence in the prediction that therespective row belongs to the group specified.

According to another embodiment of method 1421, the column passed withthe GROUP command term provides the context of a latent structure inwhich the one or more rows of each specified group are assessed forsimilarity to any other rows within the same group.

According to another embodiment of method 1421, the predictive recordset having a plurality of groups specified therein includes a listing ofrow identifiers from the dataset or the indices and a correspondingconfidence indicator for each of the row identifiers specified.

According to another embodiment of method 1421, each row corresponds toa registered voter and in which the groupings specified by thepredictive record define naturally targetable voting blocs with eachvoting bloc predicted to be likely to react similarly to a commoncampaign message, a common campaign issue, and/or common campaignadvertising.

According to another embodiment of method 1421, each row corresponds toa economic market participant and in which the groupings specified bythe predictive record define naturally targetable advertising groupswith economic market participants of each advertising group predicted toreact similarly to a common advertising campaign directed thereto.

According to another embodiment, method 1421 further includes:indicating a most representative row within each of the respectivegroups returned with the predictive record set, in which the mostrepresentative row for each of the groups returned corresponds to anactual row of the dataset.

According to another embodiment, method 1421 further includes:indicating a most stereotypical row within each of the respective groupsreturned with the predictive record set, in which the most stereotypicalrow does not exist as a row of the dataset, the most stereotypical rowhaving synthesized data based on actual rows within the dataset for thespecified group to which the most stereotypical row corresponds.

According to another embodiment, method 1421 further includes: passing aminimum confidence threshold with the GROUP command term; and in whichreturning the predictive record set includes returning only rows of thegroups in the predictive record set having a confidence indicator inexcess of the minimum confidence threshold.

According to another embodiment of method 1421, exposing the database ofthe host organization includes exposing a Predictive Query LanguageApplication Programming Interface (PreQL API) directly to authenticatedusers, in which the PreQL API is accessible to the authenticated usersvia a public Internet.

According to another embodiment of method 1421, querying the databaseusing the GROUP command term includes passing a PreQL query to thedatabase, the PreQL query having a query syntax of: the GROUP commandterm as a required term; a COLUMN term as a required term, the COLUMNterm specifying the column to be passed with the GROUP command term; andan optional FROM term specifying one or more tables, datasets, datasources, and/or indices to be queried when the optional FROM term isspecified and in which a default value is used for the one or moretables, datasets, data sources, and/or indices to be queried when theoptional FROM term is not specified.

According to another embodiment of method 1421, the query syntax for thePreQL query further provides: an optional CONFIDENCE term that, whenprovided, specifies the minimum acceptable confidence indicator for therows to be returned with the groups of the predictive record set.

According to another embodiment of method 1421, querying the databaseusing the GROUP command term includes passing a JavaScript ObjectNotation (JSON) structured query to the database, the JSON structuredquery having a query syntax of: the GROUP command term as a requiredterm; an optional one or more tables, datasets, data sources, and/orindices to be queried or a default value for the one or more tables,datasets, data sources, and/or indices to be queried when not specified,the column to be passed with the GROUP command term; and an optionalspecification of a minimum acceptable confidence for the rows of thegroups to be returned with the predictive record set according to aconfidence indicator corresponding to each of the rows.

According to another embodiment of method 1421, exposing the database ofthe host organization includes exposing a web form directly toauthenticated users, in which the web form is accessible to theauthenticated users via a public Internet; in which the hostorganization generates a latent structure query for submission to thedatabase based on input from the web form; and in which querying thedatabase using the GROUP command term includes querying the databaseusing the latent structure query via a Predictive Query LanguageApplication Programming Interface (PreQL API) within the hostorganization, the PreQL API being indirectly exposed to authenticatedusers through the web form.

According to another embodiment of method 1421, querying the databaseusing the GROUP command term includes executing a Predictive QueryLanguage (PreQL) structured query against the database for the GROUPcommand term; and in which the method further includes executing one ormore additional PreQL structured queries against the database, each ofthe one or more additional PreQL structured queries specifying at leastone command selected from the group of PreQL commands including:PREDICT, GROUP, GROUP, and GROUP.

According to another embodiment, method 1421 further includes: receivingthe dataset from an authenticated subscriber and subsequently receivingthe query for the database from the authenticated subscriber; andprocessing the dataset on behalf of the authenticated subscriber togenerate the indices.

According to another embodiment of method 1421, each of the plurality ofrows in the dataset corresponds to an entity; in which each of theplurality of columns corresponds to a characteristic for the entities;and in which a point of intersection between each respective row andeach of the plurality of columns forms a cell to store a value at thepoint of intersection.

According to another embodiment, there is a non-transitory computerreadable storage medium having instructions stored thereon that, whenexecuted by a processor in a host organization, the instructions causethe host organization to perform operations including: generatingindices from a dataset of columns and rows, the indices representingprobabilistic relationships between the rows and the columns of thedataset; storing the indices within a database of the host organization;exposing the database of the host organization via a request interface;receiving, at the request interface, a query for the database specifyinga GROUP command term and a specified column as a parameter for the GROUPcommand term; querying the database using the GROUP command term andpassing the specified column to generate a predictive record set; andreturning the predictive record set responsive to the query, thepredictive record set having a plurality of groups specified therein,each of the returned groups of the predictive record set including agroup of one or more rows of the dataset.

FIG. 15A illustrates usage of the SIMILAR command term in accordancewith the described embodiments. Using the SIMILAR command term users canask: “Which rows are most similar to a given row?” Rows can be similarin one context but dissimilar in another. For instance, killer whalesand blue whales are a lot alike in some respects, but very different inothers. Input 1501 specifies both a Row ID and a Column ID to be passedwith the SIMILAR command term. The input column (or column ID) providesthe context of the latent structure in which the specified row is to beassessed for similarity to the similar rows returned by the elements ofthe predictive record set. Responsive to such a query, a predictivedataset will be returned as output 1502 identifying how often each rowappears in the same category as the input row in the view containing theinput column.

The SIMILAR command term functionality accepts an entity (e.g., row orrow ID) and then returns what other rows are most similar to the rowspecified. Like the RELATED command term examples, the SIMILAR commandterm functionality returns the probability that a row specified and anyrespective returned row actually exhibits similarity. For instance,rather than specifying column, a user may specify “Fred” as a row orentity within the dataset. The user then queries via the SIMILAR commandterm functionality: “What rows are scored based on probability to be themost like Fred?” The API call will then return all rows from the datasetalong with corresponding confidence scores or return only rows above orbelow a specified threshold. For instance, perhaps rows above 0.8 arethe most interesting or the rows below 0.2 are most interesting, orboth, or a range. Regardless, the SIMILAR command term functionality iscapable of scoring every row in the dataset according to itsprobabilistic similarity to the specified row, and then returning therows and their respective scores according to the user's constraints orthe constraints of an implementing GUI, if any such constraints aregiven.

Because the analysis engine determines these relationships using its ownmodeling, there is more than one way to evaluate for such an inquiry.Thus, in addition to accepting the entity (e.g., row or row ID) beingassessed for similarity, the user also provides to the API call for theSIMILAR command term which COLUMN (or column ID) is to be used by theanalysis engine as a disambiguation means to determine how the row'ssimilarity is to be assessed. Thus, API calls specifying the SIMILARcommand term require both a row and a column to be fixed. In such a way,providing, specifying, or fixing the column variable providesdisambiguation information to the analysis engine by which to enter theindices. Otherwise there may be too many possible ways to score thereturned rows as the analysis engine would lack focus or an entry pointby which to determine how the user presenting the query cares about theinformation for which similarity is sought.

FIG. 15B depicts an exemplary architecture in accordance with describedembodiments. In particular, customer organizations 1505A, 1505B, and1505C are depicted, each with a client device 1506A, 1506B, and 1506Ccapable of interfacing with host organization 1510 via network 1525,including sending queries and receiving responses. Within hostorganization 1510 is a request interface 1576 which may optionally beimplemented by web-server 1575. The host organization further includesprocessor(s) 1581, memory 1582, a query interface 1580, analysis engine1585, and a multi-tenant database system 1530. Within the multi-tenantdatabase system 1530 are execution hardware, software, and logic 1520that are shared across multiple tenants of the multi-tenant databasesystem 1530, authenticator 1598, and a predictive database 1550 capableof storing indices generated by the analysis engine to facilitate thereturn of predictive record sets responsive to queries executed againstthe predictive database 1550.

According to one embodiment, the host organization 1510 operates asystem 1511 having at least a processor 1581 and a memory 1582 therein,in which the system 1511 includes an analysis engine 1585 to generateindices from a dataset of columns and rows, in which the indicesrepresent probabilistic relationships between the rows and the columnsof the dataset. Such a system 1511 further includes the predictivedatabase 1550 to store the indices; a request interface 1576 to exposethe predictive database, for example, to users or to the client devices1506A-C, in which the request interface 1576 is to receive a query 1553for the predictive database 1550 specifying a SIMILAR command term, aspecified row as a parameter for the SIMILAR command term, and aspecified column as a parameter for the SIMILAR command term. In such asystem, a query interface 1580 is to query the predictive database 1550using the SIMILAR command term and pass the specified row and thespecified column to generate a predictive record set. For instance, theSIMILAR command term and its operands (column ID and row ID) may beexecuted against the predictive database 1550.

In such a system 1511, the request interface 1576 is to further returnthe predictive record set 1554 responsive to the query 1553, in whichthe predictive record set 1554 includes a plurality of elements 1599,each of the returned elements of the predictive record set 1554including (i) a row identifier which corresponds to a row of the datasetassessed to be similar, according to a latent structure, to thespecified row passed with the SIMILAR command term based on thespecified column and (ii) a confidence indicator which indicates alikelihood of a latent relationship between the specified row passedwith the SIMILAR command and the row identifier returned for therespective element 1599.

FIG. 15C is a flow diagram illustrating a method in accordance withdisclosed embodiments. Method 1521 may be performed by processing logicthat may include hardware (e.g., circuitry, dedicated logic,programmable logic, microcode, etc.), software (e.g., instructions runon a processing device to perform various operations such transmitting,sending, receiving, executing, generating, calculating, storing,exposing, querying, processing, etc., in pursuance of the systems,apparatuses, and methods for implementing a SIMILAR command with apredictive query interface, as described herein. For example, hostorganization 110 of FIG. 1, machine 400 of FIG. 4, or system 1511 ofFIG. 15B may implement the described methodologies. Some of the blocksand/or operations listed below are optional in accordance with certainembodiments. The numbering of the blocks presented is for the sake ofclarity and is not intended to prescribe an order of operations in whichthe various blocks must occur.

At block 1591, processing logic generates indices from a dataset ofcolumns and rows, the indices representing probabilistic relationshipsbetween the rows and the columns of the dataset.

At block 1592, processing logic stores the indices within a database ofthe host organization.

At block 1593, processing logic exposes the database of the hostorganization via a request interface.

At block 1594, processing logic receives, at the request interface, aquery for the database specifying a SIMILAR command term, a specifiedrow as a parameter for the SIMILAR command term, and a specified columnas a parameter for the SIMILAR command term.

At block 1595, processing logic queries the database using the SIMILARcommand term and passes the specified row and the specified column togenerate a predictive record set.

At block 1596, processing logic returns the predictive record setresponsive to the query, the predictive record set having a plurality ofelements therein, each of the returned elements of the predictive recordset including (i) a row identifier which corresponds to a row of thedataset assessed to be similar, according to a latent structure, to thespecified row passed with the SIMILAR command term based on thespecified column and (ii) a confidence indicator which indicates alikelihood of a latent relationship between the specified row passedwith the SIMILAR command and the row identifier returned for therespective element.

According to another embodiment of method 1521, the column passed withthe SIMILAR command term provides the context of the latent structure inwhich the specified row is assessed for similarity, according to thelatent structure, to the similar rows returned by the elements of thepredictive record set.

According to another embodiment of method 1521, the row of the datasetassessed to be similar included with each element returned within thepredictive record set is based further on a fraction of times thesimilar row occurs in a same row grouping as the specified row accordingto the column passed with the SIMILAR command term.

According to another embodiment of method 1521, querying the databaseusing the SIMILAR command term and passing the specified row includespassing in a row identifier for the specified row from the dataset orthe indices.

According to another embodiment of method 1521, querying the databaseusing the SIMILAR command term and passing the specified row includespassing a complete row with the SIMILAR command term listing name=valuepairs corresponding to all columns for other rows in the dataset or theindices.

According to another embodiment of method 1521, passing the complete rowincludes passing one or more null or blank values as the value in thename=value pairs.

According to another embodiment, method 1521 further includes: returningone of: (i) a most similar row compared to the specified row passed withthe SIMILAR command term responsive to the query based on the predictiverecord set returned and a confidence indicator for each of the similarrows returned with the predictive record set; (ii) a least similar rowcompared to the specified row passed with the SIMILAR command termresponsive to the query based on the predictive record set returned anda confidence indicator for each of the similar rows returned with thepredictive record set; and (iii) a related product in a recommendersystem responsive to a search by an Internet user, in which the relatedproduct corresponds to the one of the similar rows returned with thepredictive record set.

According to another embodiment of method 1521, querying the databaseusing the SIMILAR command term includes the database estimating mutualinformation based at least in part on the specified row to determine ameasure of mutual dependence between the value of the specified row inthe indices and a value of another row present within the indices andcorresponding to the column specified.

According to another embodiment of method 1521, the rows of the datasetcorrespond to a plurality of documents stored as records in the datasetfrom which the indices are generated; in which passing the specified rowincludes passing one of the plurality of documents as the specified row;and in which querying the database using the SIMILAR command term andpassing the document as the specified row causes the database to carryout a content based search using the document's contents.

According to another embodiment, method 1521 further includes: passing aminimum confidence threshold with the SIMILAR command term; and in whichreturning the predictive record set includes returning only the elementsof the predictive record set having a confidence indicator in excess ofthe minimum confidence threshold.

According to another embodiment, method 1521 further includes: passingan optional COUNT term that, when provided, specifies a maximum quantityof elements to be returned within the predictive record set.

According to another embodiment of method 1521, the elements of thepredictive record set are returned ordered by descending order accordingto a confidence indicator for each of the elements of the predictiverecord set or are returned ordered by ascending order according to theconfidence indicator for each of the elements of the predictive recordset.

According to another embodiment, method 1521 further includes:identifying one or more of the similar rows from the predictive recordset generated responsive to the querying the database using the SIMILARcommand term based on a minimum confidence threshold for the similarrows; and inputting the identified one or more of the similar rows intoa second query specifying a GROUP command term to restrict a secondpredictive record set returned from the second query.

According to another embodiment of method 1521, exposing the database ofthe host organization includes exposing a Predictive Query LanguageApplication Programming Interface (PreQL API) directly to authenticatedusers, in which the PreQL API is accessible to the authenticated usersvia a public Internet.

According to another embodiment of method 1521, querying the databaseusing the SIMILAR command term includes passing a PreQL query to thedatabase, the PreQL query having a query syntax of: the SIMILAR commandterm as a required term; a ROW term as a required term, the ROW termspecifying the row to be passed with the SIMILAR command term; a COLUMNterm as a required term, the COLUMN term specifying the column to bepassed with the SIMILAR command term; and an optional FROM termspecifying one or more tables, datasets, data sources, and/or indices tobe queried when the optional FROM term is specified and in which adefault value is used for the one or more tables, datasets, datasources, and/or indices to be queried when the optional FROM term is notspecified.

According to another embodiment of method 1521, the query syntax for thePreQL query further provides one or more of: an optional CONFIDENCE termthat, when provided, specifies the minimum acceptable confidenceindicator to be returned with the predictive record set; an optionalCOUNT term that, when provided, specifies a maximum quantity of elementsto be returned within the predictive record set; and an optional ORDERBY term that, when provided, specifies whether the elements of thepredictive record are to be returned in ascending or descending orderaccording to a confidence indicator for each of the elements returnedwith the predictive record set.

According to another embodiment of method 1521, querying the databaseusing the SIMILAR command term includes passing a JavaScript ObjectNotation (JSON) structured query to the database, the JSON structuredquery having a query syntax of: the SIMILAR command term as a requiredterm; an optional one or more tables, datasets, data sources, and/orindices to be queried or a default value for the one or more tables,datasets, data sources, and/or indices to be queried when not specified;the row to be passed with the SIMILAR command term; in which the columnis to be passed with the SIMILAR command term; an optional specificationof a minimum acceptable confidence to be returned with the predictiverecord set according to a confidence indicator; an optionalspecification of a maximum quantity of elements to be returned withinthe predictive record set; and in which an optional specification ofwhether the elements of the predictive record are to be returned inascending or descending order according to a confidence indicator foreach of the elements returned with the predictive record set.

According to another embodiment of method 1521, exposing the database ofthe host organization includes exposing a web form directly toauthenticated users, in which the web form is accessible to theauthenticated users via a public Internet; in which the hostorganization generates a latent structure query for submission to thedatabase based on input from the web form; and in which querying thedatabase using the SIMILAR command term includes querying the databaseusing the latent structure query via a Predictive Query LanguageApplication Programming Interface (PreQL API) within the hostorganization, the PreQL API being indirectly exposed to authenticatedusers through the web form.

According to another embodiment of method 1521, querying the databaseusing the SIMILAR command term includes executing a Predictive QueryLanguage (PreQL) structured query against the database for the SIMILARcommand term; and in which the method further includes executing one ormore additional PreQL structured queries against the database, each ofthe one or more additional PreQL structured queries specifying at leastone command selected from the group of PreQL commands including:PREDICT, SIMILAR, SIMILAR, and GROUP.

According to another embodiment, method 1521 further includes: receivingthe dataset from an authenticated subscriber and subsequently receivingthe query for the database from the authenticated subscriber; andprocessing the dataset on behalf of the authenticated subscriber togenerate the indices.

According to another embodiment of method 1521, each of the plurality ofrows in the dataset corresponds to an entity; in which each of theplurality of columns corresponds to a characteristic for the entities;and in which a point of intersection between each respective row andeach of the plurality of columns forms a cell to store a value at thepoint of intersection.

According to another embodiment, there is a non-transitory computerreadable storage medium having instructions stored thereon that, whenexecuted by a processor in a host organization, the instructions causethe host organization to perform operations including: generatingindices from a dataset of columns and rows, the indices representingprobabilistic relationships between the rows and the columns of thedataset; storing the indices within a database of the host organization;exposing the database of the host organization via a request interface;receiving, at the request interface, a query for the database specifyinga SIMILAR command term, a specified row as a parameter for the SIMILARcommand term, and a specified column as a parameter for the SIMILARcommand term; querying the database using the SIMILAR command term andpassing the specified row and the specified column to generate apredictive record set; and returning the predictive record setresponsive to the query, the predictive record set having a plurality ofelements therein, each of the returned elements of the predictive recordset including (i) a row identifier which corresponds to a row of thedataset assessed to be similar, according to a latent structure, to thespecified row passed with the SIMILAR command term based on thespecified column and (ii) a confidence indicator which indicates alikelihood of a latent relationship between the specified row passedwith the SIMILAR command and the row identifier returned for therespective element.

FIG. 16A illustrates usage of the PREDICT command term in accordancewith the described embodiments. More particularly, the embodiment shownillustrates use of classification and/or regression to query the indicesusing the PREDICT command term in which the input 1601 to the PREDICTcommand term fixes a subset of the columns and further in which theoutput 1602 predicts a single target column. As can be seen from theexample, the left most column is to be predicted (e.g., the output 1602of the PREDICT command term) and several columns are provided to thePREDICT command term as input 1601 (e.g., fifth, seventh, eight,eleventh, twelfth, thirteenth, and sixteenth columns).

With the cross-categorizations technique having been used to create theindices a prediction request presented via the PREDICT command term istreated as a new row for the dataset and the analysis engine assignsthat new row to categories in each cross-categorization. Next, using theselected standardized distributions for each category, the valuesrequested are predicted. Unlike conventional predictive analytics, theanalysis engine and use of the PREDICT command term provides forflexible predictive queries without customized implementation of modelsspecific to the dataset being analyzed, thus allowing the user of thePREDICT command term to specify as many or as few columns as they desireand further allowing the analysis engine to predict as many or as fewelements according to the user's request.

For instance, consider classification or regression in which all but oneof the columns are used to predict a single target column. The analysisengine can render the prediction using a single target column or canrender the prediction using a few target columns at the user'sdiscretion. For instance, certain embodiments permit a user to query theindices via the PREDICT command term to ask a question such as: “Will anopportunity close AND at what amount?” Such capabilities do not existwithin conventionally available means.

Using the PREDICT command term, calling an appropriate API for thePREDICT functionality enables users to predict any chosen sub-set ofdata to predict any column or value. It is not required that an entiredataset be utilized to predict only a single value, as is typical withcustom implemented models.

When using the PREDICT command term, the user provides or fixes thevalue of any column and then the PREDICT API call accepts the fixedvalues and those the user wants to predict. The PREDICT command termfunctionality then queries the indices (e.g., via the analysis engine orthrough the PreQL interface or query interface, etc.) asking: “Given arow that has these values fixed, as provided by the user, then what willthe distribution be?” For instance, the functionality may fix all butone column in the dataset and then predict the last one, the missingcolumn, as is done with customized models. But the PREDICT command termfunctionality is far more flexible than conventional models that arecustomized to a specific dataset. For instance, a user can change thecolumn to be predicted at a whim whereas custom implemented modelssimply lack this functionality as they lack the customized mathematicalconstructs to predict for such unforeseen columns or inquiries. That isto say, absent a particular function having been pre-programmed, theconventional models simply cannot perform this kind of varying querybecause conventional models are hard-coded to solve for a particularcolumn. Conversely, the methodologies described herein are nothard-coded or customized for any particular column or dataset, and assuch, a user is enabled to explore their data by making multipledistinct queries or adapt their chosen queries simply by changing thecolumns to be predicted as their business needs change over time even ifthe underlying data and data structures of the client organization donot remain constant.

Perhaps also the user does not know all the columns to fix. Forinstance, the dataset may contain only limited observed values about oneuser yet have lots of data about another user. For instance, anecommerce site may know little about a non-registered passerby user butknows lots of information about a registered user with a rich purchasehistory. In such an example, the PREDICT command term functionalitypermits fixing or filling in only the stuff that is known without havingto require all the data for all users, as some of the data is known tobe missing, and thus, the PREDICT command term easily accommodatesmissing data and null values that exist in a user's real-world data set.In such a way, the PREDICT command term functionality can still predictmissing data elements using the data that is actually known.

Another capability using the PREDICT command term functionality is tospecify or fix all the data in a dataset that is known, that is, allnon-null values, and then fill in everything else. In such a way, a usercan say that what is observed in the dataset is known, and for the datathat is missing, render predictions. The PREDICT functionality will thusincrease the percentage of filled or completed data in a dataset byutilizing predicted data for missing or null-values by acceptingpredictions having a predictive quality over a user's specifiedconfidence, or accept all predicted values by sufficiently lowering theminimum confidence threshold required by the user. This functionality isalso implemented by a specialized GUI interface as is described herein.

Another functionality using PREDICT is to fill in an empty set. So maybedata is wholly missing for a particular entity row (or rows), and usingthe PREDICT command term functionality, synthetic data may be generatedthat represents new rows with the new data in those rows representingplausible, albeit synthetic data.

In other embodiments, PREDICT can be used to populate data elements thatare not known but should be present or may be present, yet are notfilled in within the data set, thus allowing the PREDICT functionalityto populate such data elements.

Another example is to use PREDICT to attain a certainty or uncertaintyfor any element and to display or return the range of plausible valuesfor the element.

FIG. 16B illustrates usage of the PREDICT command term in accordancewith the described embodiments. More particularly, the embodiment shownillustrates use of a “fill-in-the-blanks” technique in which missingdata or null values within a tabular dataset are filled with predictedvalues by querying previously generated indices using the PREDICTcommand term in which the input 1611 to the PREDICT command term fixes asubset of the columns and further in which the output 1612 predicts allof the missing columns or missing elements (e.g., null values) withinthe remaining missing columns.

For example, a user can take an incomplete row (such as the topmost rowdepicted with the numerous question marks) and via the PREDICT commandterm, the user can predict all of the missing values to fill in theblanks. At the extreme, the user can specify as the dataset to beanalyzed a table with many missing values across many rows and manycolumns and then via the PREDICT command term the user can render atable where all of the blanks have been filled in with valuescorresponding to varying levels of confidence quality.

Specialized tools for this particular use case are discussed below inwhich UI functionality allows the user to trade off confidence quality(e.g., via a confidence score or a confidence indicator) to populatemore or less data within such a table such that more data (or all thedata) can be populated by degrading confidence or in the alternative,some but not all can be populated, above a given confidence qualitythreshold which is configurable by the user, and so forth. A use casespecialized GUI is additionally provided and described for thisparticular use case in more detail below. According to certainembodiments, such a GUI calls the PREDICT command term via an API onbehalf of the user, but nevertheless utilizes the analysis engine'sfunctional core consistent with the methodologies described herein toissue PREDICT command term based PreQL queries.

FIG. 16C illustrates usage of the PREDICT command term in accordancewith the described embodiments. More particularly, the embodiment shownillustrates use of synthetic data generation techniques in which datathat is not actually present within any column or row of the originaldataset, but is nevertheless consistent with the original dataset, isreturned as synthetic data. Synthetic data generation again utilizes thePREDICT command term as the only input 1621 with none of the columnsbeing fixed. Output 1622 results in all of the columns being predictedfor an existing dataset rendering a single synthetic row or renderingmultiple synthetic rows, as required by the user.

Such functionality may thus be utilized to fill in an empty set as theoutput 1622 by calling the PREDICT command term with no fixed columns asthe input 1621. Take for example, an entity, real or fictitious, forwhich the entity row data is wholly missing. By querying the indicesusing the PREDICT command term the analysis engine will generate datathat represents the empty set by providing new entity rows in which thegenerated synthetic data within the rows provides plausible data, albeitsynthetic data. That is to say, the predicted values for such rows arenot pulled from the dataset as actually observed data but neverthelessrepresents data that plausibly may have been observed within thedataset. A confidence quality indicator may, as before, also be utilizedto better tune the output 1622 to the user's particular needs.

The synthetic row generated by the analysis engine responsive to thePREDICT command term call will output 1622 one or more entity rows thatexhibit all of the structure and predictive relationships as are presentin the real data actually observed and existing within the datasetanalyzed by the analysis engine. Such a capability may enable a user togenerate and then test a dataset that is realistic, but in no waycompromises real-world data of actual individuals represented by theentity rows in the dataset without forcing the user seeking such data tomanually enter or guess at what such data may look like. This may behelpful in situations where a dataset is needed for test purposesagainst very sensitive information such as financial data forindividuals, HIPAA (Health Insurance Portability and Accountability Act)protected health care data for individuals, and so forth.

FIG. 16D depicts an exemplary architecture in accordance with describedembodiments. In particular, customer organizations 1605A, 1605B, and1605C are depicted, each with a client device 1606A, 1606B, and 1606Ccapable of interfacing with host organization 1610 via network 1625,including sending queries and receiving responses. Within hostorganization 1610 is a request interface 1676 which may optionally beimplemented by web-server 1675. The host organization further includesprocessor(s) 1681, memory 1682, a query interface 1680, analysis engine1685, and a multi-tenant database system 1630. Within the multi-tenantdatabase system 1630 are execution hardware, software, and logic 1620that are shared across multiple tenants of the multi-tenant databasesystem 1630, authenticator 1698, and a predictive database 1650 capableof storing indices generated by the analysis engine to facilitate thereturn of predictive record sets responsive to queries executed againstthe predictive database 1650.

According to one embodiment, the host organization 1610 operates asystem 1631 having at least a processor 1681 and a memory 1682 therein,in which the system 1631 includes an analysis engine 1685 to generateindices from a dataset of columns and rows, in which the indicesrepresent probabilistic relationships between the rows and the columnsof the dataset. Such a system 1631 further includes the predictivedatabase 1650 to store the indices; a request interface 1676 to exposethe predictive database, for example, to users or to the client devices1606A-C, in which the request interface 1676 is to receive a query 1653for the database 1650 specifying at least (i) a PREDICT command term,(ii) one or more specified columns to be predicted, and (iii) one ormore column name=value pairs specifying column names to be fixed and thevalues by which to fix them. According to such a system, a queryinterface 1680 is to query 1653 the predictive database 1650 using thePREDICT command term and passing the one or more specified columns to bepredicted and the one or more column name=value pairs to generate arepresentation of a joint conditional distribution of the one or morespecified columns to be predicted fixed according to the columnname=value pairs using the indices stored in the database 1650. Forinstance, the PREDICT command term and its operands (the one or morecolumn IDs and the one or more column name=value pairs) may be executedagainst the predictive database 1650.

In such a system 1631, the request interface 1676 is to further returnthe representation of a joint conditional distribution of the one ormore specified columns as output 1654 responsive to the query 1653.

FIG. 16E is a flow diagram 1632 illustrating a method in accordance withdisclosed embodiments. Method 1632 may be performed by processing logicthat may include hardware (e.g., circuitry, dedicated logic,programmable logic, microcode, etc.), software (e.g., instructions runon a processing device to perform various operations such transmitting,sending, receiving, executing, generating, calculating, storing,exposing, querying, processing, etc., in pursuance of the systems,apparatuses, and methods for implementing a PREDICT command with apredictive query interface, as described herein. For example, hostorganization 110 of FIG. 1, machine 400 of FIG. 4, or system 1631 ofFIG. 16D may implement the described methodologies. Some of the blocksand/or operations listed below are optional in accordance with certainembodiments. The numbering of the blocks presented is for the sake ofclarity and is not intended to prescribe an order of operations in whichthe various blocks must occur.

At block 1691, processing logic generates indices from a dataset ofcolumns and rows, the indices representing probabilistic relationshipsbetween the rows and the columns of the dataset.

At block 1692, processing logic stores the indices within a database ofthe host organization.

At block 1693, processing logic exposes the database of the hostorganization via a request interface.

At block 1694, processing logic receives, at the request interface, aquery for the database specifying at least (i) a PREDICT command term,(ii) one or more specified columns to be predicted, and (iii) one ormore column name=value pairs specifying column names to be fixed and thevalues by which to fix them.

For instance, the one or more column name=value pairs passed with thePREDICT command term according to (iii) above may take the form of, byway of example only, column_abc=‘string_xyz’ or alternatively,{column_abc=‘string_xyz’} or alternatively, column_abc|‘string_xyz’ andso forth. Other syntax is permissible according to the API and/or queryinterface accepting the query. Moreover, multiple such column name=valuepairs may be passed.

At block 1695, processing logic queries the database using the PREDICTcommand term and passes the one or more specified columns to bepredicted and the one or more column name=value pairs to generate arepresentation of a joint conditional distribution of the one or morespecified columns to be predicted fixed according to the columnname=value pairs using the indices stored in the database.

Processing logic may additionally return the representation of a jointconditional distribution of the one or more specified columns as output,for instance, within a predictive record set responsive to the query.

According to another embodiment, method 1632 further includes:generating a predictive record set responsive to the querying; in whichthe predictive record set includes a plurality of elements therein, eachof the elements specifying a value for each of the one or more specifiedcolumns to be predicted; and in which the method further includesreturning the predictive record set responsive to the query.

According to another embodiment of method 1632, exposing the database ofthe host organization includes exposing a Predictive Query LanguageApplication Programming Interface (PreQL API) directly to authenticatedusers, in which the PreQL API is accessible to the authenticated usersvia a public Internet.

According to another embodiment of method 1632, querying the databaseusing the PREDICT command term includes passing a PreQL query to thedatabase, the PreQL query having a query syntax of: the PREDICT commandterm as a required term; a required TARGET term specifying the one ormore specified columns to be predicted; a required WHERE term thatspecifies the column names to be fixed and the values by which to fixthem as the one or more column name=value pairs, in which the requiredWHERE term restricts output of the query to a predictive record sethaving returned elements that are probabilistically related to the oneor more columns to be fixed and the corresponding values by which to fixthem as specified; and an optional FROM term specifying one or moretables, datasets, data sources, and/or indices to be queried, when theoptional FROM term is specified.

According to another embodiment of method 1632, querying the databaseusing the PREDICT command term includes passing a JavaScript ObjectNotation (JSON) structured query to the database, the JSON structuredquery having a query syntax of: the PREDICT command term as a requiredterm; required specification of the one or more specified columns to bepredicted; required specification of the column names to be fixed andthe values by which to fix them as the one or more column name=valuepairs restricting output of the query to a predictive record set havingreturned elements that are probabilistically related to the one or morecolumns to be fixed and the corresponding values by which to fix them asspecified via the one or more column name=value pairs; and an optionalspecification of one or more tables, datasets, data sources, and/orindices to be queried.

According to another embodiment of method 1632, exposing the database ofthe host organization includes exposing a web form directly toauthenticated users, in which the web form is accessible to theauthenticated users via a public Internet.

According to another embodiment of method 1632, the host organizationgenerates a predictive query for submission to the database based oninput from the web form; and in which querying the database using thePREDICT command term includes querying the database using the predictivequery via a Predictive Query Language Application Programming Interface(PreQL API) within the host organization, the PreQL API being exposedindirectly to the authenticated users through the web form.

According to another embodiment, method 1632 further includes: returninga predictive record set specifying a predicted value for each of thecolumns originally in the dataset.

According to another embodiment, method 1632 further includes: returninga synthetic data set responsive to the querying, in which the syntheticdata includes synthetic rows having data therein which is consistentwith the rows and the columns originally with the dataset according tothe indices' probabilistic relationships between the rows and thecolumns but does not include any original record of the dataset.

According to another embodiment of method 1632, returning the syntheticdataset includes at least one of: anonymizing financial records from thedataset; anonymizing medical records from the dataset; and anonymizingInternet user records from the dataset.

According to another embodiment, method 1632 further includes: returningdistributions based on the probabilistic relationships between the rowsand the columns of the dataset using the indices; and in which thedistributions returned include synthetic data from the indices which aremathematically derived from the columns and rows of the dataset butcontain information about data that was not in any original record ofthe dataset and further in which the indices from which thedistributions are derived are not constrained to the scope of the dataof the original records of the dataset.

According to another embodiment, method 1632 further includes returningat least one of: a confidence score for the distributions, in which theconfidence score ranges from 0 to 1 with 0 indicating no confidence inthe predicted value and with 1 indicating a highest possible confidencein the predicted value; and confidence intervals indicating a minimumand maximum value between which there is a certain confidence a valuelies.

According to another embodiment of method 1632, returning thedistributions based on the probabilistic relationships, furtherincludes: passing an optional record count term with the PREDICT commandterm when querying the database, the optional record count termspecifying a quantity of records to be returned responsive to thequerying; and determining a required quantity of processing resourcesnecessary to return the quantity of records specified by the recordcount.

According to another embodiment of method 1632, returning thedistributions based on the probabilistic relationships, furtherincludes: passing a minimum accuracy threshold with the PREDICT commandterm when querying the database; and determining a required populationof samples to be returned to satisfy the minimum accuracy threshold as alower bound.

According to another embodiment of method 1632, querying the databaseusing the PREDICT command term includes executing a Predictive QueryLanguage (PreQL) structured query against the database for the PREDICTcommand term; and in which the method further includes executing one ormore additional PreQL structured queries against the database, each ofthe one or more additional PreQL structured queries specifying at leastone command selected from the group of PreQL commands including:PREDICT, RELATED, SIMILAR, and GROUP.

According to another embodiment, method 1632 further includes: receivingthe dataset from an authenticated subscriber and subsequently receivingthe query for the database from the authenticated subscriber; andprocessing the dataset on behalf of the authenticated subscriber togenerate the indices.

According to another embodiment of method 1632, each of the plurality ofrows in the dataset corresponds to an entity; in which each of theplurality of columns corresponds to a characteristic for the entities;and in which a point of intersection between each respective row andeach of the plurality of columns forms a cell to store a value at thepoint of intersection.

According to another embodiment there is a non-transitory computerreadable storage medium having instructions stored thereon that, whenexecuted by a processor in a host organization, the instructions causethe host organization to perform operations including: generatingindices from a dataset of columns and rows, the indices representingprobabilistic relationships between the rows and the columns of thedataset; storing the indices within a database of the host organization;exposing the database of the host organization via a request interface;receiving, at the request interface, a query for the database specifyingat least (i) a PREDICT command term, (ii) one or more specified columnsto be predicted, and (iii) one or more column name=value pairsspecifying column names to be fixed and the values by which to fix them;and querying the database using the PREDICT command term and passing theone or more specified columns to be predicted and the one or more columnname=value pairs to generate a representation of a joint conditionaldistribution of the one or more specified columns to be predicted fixedaccording to the column name=value pairs using the indices stored in thedatabase.

FIG. 16F depicts an exemplary architecture in accordance with describedembodiments. The embodiment depicted here is identical to that of FIG.16D except that the query 1657 specifying the PREDICT command term isutilized with zero columns fixed, that is, there are no column IDspassed with the PREDICT command term whatsoever. Consequently, theoutput 1658 returned responsive to the query 1657 provides syntheticdata generated having one or more entity rows with predicted values forevery column of the dataset.

According to one embodiment, the host organization 1610 operates asystem 1635 having at least a processor 1681 and a memory 1682 therein,in which the system 1635 includes an analysis engine 1685 to generateindices from a dataset of columns and rows, in which the indicesrepresent probabilistic relationships between the rows and the columnsof the dataset. Such a system 1635 further includes the predictivedatabase 1650 to store the indices; a request interface 1676 to exposethe predictive database, for example, to users or to the client devices1606A-C, in which the request interface 1676 is to receive a query 1657for the predictive database 1650 specifying a PREDICT command term andwith zero columns fixed such that no column IDs are passed with thePREDICT command term. In such a system, a query interface 1680 is toquery 1657 the predictive database 1650 using the PREDICT command termwithout any specified columns to generate as output 1658, generatedsynthetic data having one or more entity rows with predicted values forevery column of the dataset. In such a system 1611, the requestinterface 1676 is to further return the generated synthetic data havingone or more entity rows with predicted values for every column of thedataset as output 1658 responsive to the query 1657.

FIG. 16G is a flow diagram 1633 illustrating a method in accordance withdisclosed embodiments. Method 1633 may be performed by processing logicthat may include hardware (e.g., circuitry, dedicated logic,programmable logic, microcode, etc.), software (e.g., instructions runon a processing device to perform various operations such transmitting,sending, receiving, executing, generating, calculating, storing,exposing, querying, processing, etc., in pursuance of the systems,apparatuses, and methods for implementing a PREDICT command with apredictive query interface, as described herein. For example, hostorganization 110 of FIG. 1, machine 400 of FIG. 4, or system 1635 ofFIG. 16F may implement the described methodologies. Some of the blocksand/or operations listed below are optional in accordance with certainembodiments. The numbering of the blocks presented is for the sake ofclarity and is not intended to prescribe an order of operations in whichthe various blocks must occur.

At block 1670, processing logic generates indices from a dataset ofcolumns and rows, the indices representing probabilistic relationshipsbetween the rows and the columns of the dataset.

At block 1671, processing logic stores the indices within a database ofthe host organization.

At block 1672, processing logic exposes the database of the hostorganization via a request interface.

At block 1673, processing logic receives, at the request interface, aquery for the database specifying a PREDICT command term and one or morespecified columns to be passed with the PREDICT command term.

At block 1674, processing logic queries the database using the PREDICTcommand term and the one or more specified columns to generate output,in which the output includes generated synthetic data having one or moreentity rows with predicted values for every column of the dataset usingthe indices stored in the database.

Processing logic may additionally return the generated synthetic data asoutput, for instance, within a predictive record set responsive to thequery.

According to another embodiment, method 1633 further includes: returningthe generated synthetic data having one or more entity rows withpredicted values for every column of the dataset as a synthetic data setresponsive to the querying, in which the generated synthetic dataincludes synthetic rows having data therein which is consistent with therows and the columns originally with the dataset according to theindices' probabilistic relationships between the rows and the columnsbut does not include any original record of the dataset.

According to another embodiment of method 1633, returning the syntheticdataset includes at least one of: anonymizing financial records from thedataset; anonymizing medical records from the dataset; and anonymizingInternet user records from the dataset.

According to another embodiment there is a non-transitory computerreadable storage medium having instructions stored thereon that, whenexecuted by a processor in a host organization, the instructions causethe host organization to perform operations including: generatingindices from a dataset of columns and rows, the indices representingprobabilistic relationships between the rows and the columns of thedataset; storing the indices within a database of the host organization;exposing the database of the host organization via a request interface;receiving, at the request interface, a query for the database specifyinga PREDICT command term and with zero columns fixed such that no columnIDs are passed with the PREDICT command term; and querying the databaseusing the PREDICT command term and the one or more specified columns togenerate output, in which the output includes generated synthetic datahaving one or more entity rows with predicted values for every column ofthe dataset using the indices stored in the database.

FIG. 17A depicts a Graphical User Interface (GUI) 1701 to display andmanipulate a tabular dataset having missing values by exploiting aPREDICT command term. More particularly, a GUI is provided at a displayinterface to a user which permits the user to upload or specify adataset having columns and rows and then display the dataset as a tableand subject it to manipulation by populating missing values (e.g.,null-values) with predicted values. At element 1707 the user specifiesthe data to be analyzed and displayed via the GUI 1701. For instance,the user may browse a local computing device for a file, such as anexcel spreadsheet, and then upload the file to the system for analysis,or the user may alternatively specify a dataset which is accessible tothe host organization which is providing the GUI 1701. For instance, thehost organization is a cloud based service provider and where the user'sdataset already resides within the cloud, the user can simply specifythat dataset as the data source via the action at element 1707.

In the example depicted at FIG. 17A, the displayed table provided by theuser is 61% filled. The table is only partially filled because theuser's dataset provided has many missing data elements. The presentlydisplayed values in grayscale depict known values, such as the knownvalue “1.38” at the topmost row in the Proanthocyanine column depictedby element 1703. Two columns to the right at the Proline column there isa null value displayed at the topmost row displayed simply as an emptycell as depicted by element 1702.

In this initial depiction, all known values 1703 are depicted andcorrespond to actual observed data within the underlying dataset.According to this embodiment, the slider 1705 which operates as athreshold modifier is all the way to the left hand side and representsthe minimum fill 1704 given that all known values are displayed withoutany predicted values being displayed. Accordingly, the confidence of allvalues displayed may be considered to be 100% given that all values areactually observed within the dataset provided and no values arepredicted. The slider control may be utilized as a threshold modifier tocontrol the fill percentage of the table which in turn alters thenecessary confidence thresholds to attain a user specified fillpercentage or alternatively, the slider control may be utilized as athreshold modifier to control the user's acceptable level ofuncertainty, and thus, as the user's specified acceptable level ofuncertainty changes, the percentage of fill of the table will increaseor decrease according to available predictive values that comply withthe user's specified acceptable level of uncertainty. In otherembodiments, acceptable level of uncertainty may be specified via, forexample, a text entry box, or other control means.

The user may click the download 1706 action to download the displayedtable in a variety of formats, however, such a table will correspond tothe source table that was just specified or uploaded via the data 1707action.

No values are predicted, but the user may simply move the slider toincrease the data fill for the missing values, causing the GUI'sfunctionality to utilize the predict function on behalf of the user.Just above the slider the user is informed of the current state of theminimum fill 1704, which according to the example displayed, is the 61%as noted above, but will change as the slider is moved.

FIG. 17B depicts another view of the Graphical User Interface. Here, thedisplayed table has populated some but not all of the null values (e.g.,missing data) with predicted values. For instance, the previously emptycell in the topmost row at the Proline column corresponding to nullvalue within the user's underlying dataset is now populated withpredicted value “564” as depicted by element 1708. Notably, the value564 does not reside at this location within the user's underlyingdataset and was not observed within the user's underlying dataset.Rather, the GUI 1701 has instituted a PREDICTED command term call on theuser's behalf to retrieve the predicted value 1708 result displayedhere. In this example, all of the values in gray scale are known valuesand all of the values of the table displayed in solid black arepredicted values that have replaced previously unknown null values ofthe same dataset.

The slider now shows 73% fill as depicted by element 1709 and some butnot all missing values are now populated with predicted values. The filllevel is user controllable simply by moving the slider back and forth tocause the GUI 1701 to populate missing data values with predicted valuesor to remove predicted values as the user's specified acceptable levelof uncertainty is increased or decreased respectively.

Not depicted on this example is a user configurable minimum confidencethreshold which may be set via a text box, dropdown, slider, etc. Such aGUI element permits the user to specify the minimum confidence requiredfor a predicted value to be displayed at the GUI 1701. In one embodimenthaving the minimum confidence threshold additionally causes a maximumfill value to be displayed and the slider at element 1709 is thenlimited to the maximum fill as limited by the minimum confidencethreshold.

This is because as the fill percentage increases it is necessary todegrade the confidence quality of the predicted values populating thenull values. Conversely, as the fill percentage decreases the confidencequality may be increased.

Thus, if a user dictates a perfect (e.g., 100%) confidence quality, thenit is unlikely that any null values can be filled because it is unlikelyto predict with 100% confidence any missing value. All of the actuallyobserved values will, however, continue to be displayed as they areknown from the underlying dataset with 100% confidence. Conversely, ifthe same user dictates a very low confidence (e.g., 25%), then it isvery likely that most, if not all missing values can be predicted as the25% requirement is a low threshold in terms of confidence quality. It isfeasible with some datasets that all or nearly all of the null valuesmay be predicted with a relatively high (e.g., 80%) confidence dependingon the quality of the underlying dataset, the size of the population inthe underlying dataset, etc. Regardless, the GUI 1701 permits the userto experiment with their own dataset in a highly intuitive mannerwithout even having to understand how the PREDICT command term operates,what inputs it requires, how to make the PreQL or JSON API call, and soforth. Such a GUI 1701 therefore can drastically lower the learningcurve of lay users wishing to utilize the predictive capabilitiesprovided by the analysis engine's core.

FIG. 17C depicts another view of the Graphical User Interface. Here theGUI 1701 retains its previously depicted known values 1703 and predictedvalues 1708 but the user controllable slider has been moved all the wayto the right to a 100% maximum fill as depicted by element 1709, suchthat all known values remain in the table display but 100% of the nullvalues in the dataset are also populated and displayed at the GUI 1701.

Additionally shown is a minimum confidence threshold action at element1710 as an optional input field for specifying the minimum confidencethreshold that was noted previously (e.g., via a dropdown, text box,slider, radio buttons, etc.). Some tables can be displayed at 100% witha minimum confidence threshold greater than zero whereas others willrequire that if the minimum confidence threshold is specified at 1710,then it may need to be at or near zero if the underlying quality of thedataset is poor. These determinations will fall out of the datasetaccording to the probabilistic interrelatedness of the data elements andthe presence or absence of noise.

Nevertheless, the minimum confidence threshold is specified at 1710permits a lay user to experiment with their dataset in a highlyintuitive manner. If the user specifies a minimum confidence thresholdat 1710 that does not permit a 100% fill, then the max % filled or fillpercentage will indicate the extent of fill feasible according to theminimum confidence threshold set by the user at 1710 when the slider ismoved all the way to the right.

Because the table is displayed at 100% fill, all null or missing valuesare predicted, but it may be necessary to degrade the confidencesomewhat to attain the 100% fill, in which case the optional minimumconfidence threshold at 1710 may remain unset, grayed out, deactivated,or simply not displayed to the user.

According to certain embodiments, the chosen fill level or acceptablelevel of uncertainty, as selected by the user via the slider bar (orcontrolled via the optional minimum confidence threshold at 1710) can be“saved” by clicking the download action to capture the displayeddataset. The displayed copy can be saved as a new version or saved overthe original version of the table at the discretion of the user, thusresulting in the predictive values provided being saved or input to thecell locations within the user's local copy. Metadata can additionallybe used to distinguish the predicted values from actual known andobserved values such that subsequent use of the dataset is not corruptedor erroneously influenced by the user's experimental activities usingthe GUI 1701.

The control slider at element 1709 is feasible because when a user asksfor a value to be predicted, such as a missing value for “income,” whatis actually returned to the GUI functionality making the PREDICT commandterm API call is the respective persons' income distribution aspredicted by the analysis engine modeling in order to generate theindices which are then queried by the PREDICT command term. The returneddistribution for a predicted value permits the GUI to select a value tobe displayed as well as restrict the display according to confidencequality. In other embodiments, a confidence indicator is returned ratherthan a distribution.

By using such a GUI interface or such a concept in general, the user isgiven control over accuracy and confidence. In such a way, the user canmanipulate how much data is to be filled in and to what extent theconfidence quality threshold applies, if at all. Behind the scenes andout of view from the user, the GUI 1701 makes PREDICT command term APIcalls against an analyzed dataset specified by the user. The analysisengine takes the user's dataset, such as a table with a bunch of typedcolumns, and then renders a prediction for every single cell having anull value at the request of the GUI 1701. For each cell that ismissing, the GUI 1701 is returned a distribution or a confidenceindicator from the PREDICT command term API calls and when the slider ismanipulated by a user, functionality of the GUI's slider looks at thedistributions for the null values, looks at variances for thedistributions of the null values, and then displays its estimates as thepredicted values shown in the examples. Thus, for any given cell havinga predicted result in place of the missing null value, the GUI 1701 byexploiting the PREDICT command term functionality represents to the usera value for the null value on the basis of having seen multiple otherknown values or observed values in the underlying dataset. The GUI 1701itself does not perform the analysis of the dataset but merely benefitsfrom the data returned from the PREDICT command term API calls as notedabove.

According to one embodiment, starting with nothing more than raw data ina tabular form, such as data in a spreadsheet or data stored within oneor more tables of a relational database, an UPLOAD command term API callis first made by the GUI to upload or insert the data into thepredictive database upon which the analysis engine operates to analyzethe data, either automatically or responsive to an ANALYZE command termAPI call. For example, where the user is paying service fees for accessto the functionality the GUI may indicate pricing to the user uponuploading of the data and request acceptance prior to triggering theanalyzing by the analysis engine. In other instances the analysis enginesimply performs the analysis automatically. Regardless, upon uploadingthe data specified by the user, the data looks just like all othertabular data, but once uploaded and analyzed by the analysis engine, aprobabilistic model is executed against the data and the analysis enginelearns through its modeling how the rows and the columns can interactwith each other through which various probabilistic relationships andcausations are built and represented within the generated indices as isdescribed herein. For instance, a generated statistical index figuresout how and which columns are related to another to learn, for instance,that a particular subset of columns are likely to share a causal origin.

The difficult problem is that the analysis engine must perform itsanalysis using real world data provided by the user rather than pristineand perfect datasets and most do so without knowing in advance theunderlying structure of the data to be analyzed. With data that existsin the real world, some columns are junk, some columns are duplicates,some columns are heterogeneous (e.g., not consistently data typed), somecolumns are noisy with only sparsely populated data or populated withnoisy erroneous data, etc. The analysis engine through its statisticalindex and other modeling identifies the appropriate relationships andcausations despite the absence of perfectly pristine data or astandardized data structure.

Through the statistical index and other modeling, a distribution ofindices results in a model that is stored as queryable indices insupport of the predictive queries including those utilized by thedescribed GUI 1701. Other specialized GUIs and API tools which alsoutilize the PREDICT command term as well as other predictive PreQLqueries include business opportunity scoring, next best offeridentification, etc. These and other examples are described inadditional detail later in the specification.

FIG. 17D depicts an exemplary architecture in accordance with describedembodiments. In particular, customer organizations 1705A, 1705B, and1705C are depicted, each with a user's client device and display 1706A,1706B, and 1706C capable of interfacing with host organization 1710 vianetwork 1725, including sending input, queries, and requests andresponsively receiving responses including output for display. Withinhost organization 1710 is a request interface 1776 which may optionallybe implemented by web-server 1775. The host organization furtherincludes processor(s) 1781, memory 1782, a query interface 1780,analysis engine 1785, and a multi-tenant database system 1730. Withinthe multi-tenant database system 1730 are execution hardware, software,and logic 1720 that are shared across multiple tenants of themulti-tenant database system 1730, authenticator 1798, and a predictivedatabase 1750 capable of storing indices generated by the analysisengine 1785 to facilitate the return of predictive record setsresponsive to queries executed against the predictive database 1750.

According to one embodiment, the host organization 1710 operates asystem 1711 having at least a processor 1781 and a memory 1782 therein,in which the system 1711 includes a request interface 1776 to receive atabular dataset 1753 from a user as input, in which the tabular datasetincludes data values organized as columns and rows. The user may providethe tabular dataset 1753 as a file attachment or specify the locationfor the tabular dataset 1753. Such a system 1711 further includes ananalysis engine 1785 to identify a plurality of null values within thetabular dataset 1753 received from the user or specified by the user, inwhich the null values are dispersed across multiple rows and multiplecolumns of the tabular dataset. In such an embodiment, the analysisengine 1785 further generates indices 1754 from the tabular dataset ofcolumns and rows, in which the indices represent probabilisticrelationships between the rows and the columns of the tabular dataset1753. The request interface 1776 is to return the tabular dataset asdisplay output 1755 to the user, the display output 1755 including thedata values depicted as known values and the null values depicted asunknown values; the request interface 1776 is to further receive inputto populate 1756 from the user. Such input may be, for example, inputvia a slider control, a user specified minimum confidence threshold,etc. The input to populate 1756 received from the user specifies that atleast a portion of the unknown values within the displayed tabulardataset are to be populated with predicted values 1758 retrieved fromthe indices stored within the predictive database 1750. Such predictedvalues 1758 may be returned from the indices stored within thepredictive database 1750 responsive to queries 1757 constructed andissued against the predictive database 1750 by the analysis engine 1785and/or query interface 1780.

For example, in such a system the query interface 1780 may query theindices (e.g., via queries 1757) for the predicted values 1758subsequent to which the request interface 1776 returns the predictedvalues 1758 as updated display output 1759 to the user via the user'sclient device and display 1706A-C. For example, the updated displayoutput then presents at the user's client device and display 1706A-C nowdepicting predicted values in place of the previously depicted unknownvalues corresponding to missing data or null value entries within theoriginal tabular dataset 1753 provided or specified by the user.

According to another embodiment, the system 1711 further includes apredictive database 1750 to store the indices generated by the analysisengine. In such an embodiment, the predictive database 1750 is toexecute as an on-demand cloud based service at the host organization1710 for one or more subscribers.

In another embodiment, the system 1711 further includes an authenticator1798 to verify the user (e.g., a user at one of the user's client deviceand display 1706A-C) as a known subscriber. The authenticator 1798 thenfurther operates to verify authentication credentials presented by theknown subscriber.

In another embodiment, the system 1711 further includes a web-server1775 to implement the request interface; in which the web-server 1775 isto receive as input; a plurality of access requests from one or moreclient devices from among a plurality of customer organizationscommunicably interfaced with the host organization via a network; amulti-tenant database system with predictive database functionality toimplement the predictive database; and in which each customerorganization is an entity selected from the group consisting of: aseparate and distinct remote organization, an organizational groupwithin the host organization, a business partner of the hostorganization, or a customer organization that subscribes to cloudcomputing services provided by the host organization.

FIG. 17E is a flow diagram illustrating a method in accordance withdisclosed embodiments. Method 1721 may be performed by processing logicthat may include hardware (e.g., circuitry, dedicated logic,programmable logic, microcode, etc.), software (e.g., instructions runon a processing device to perform various operations such transmitting,sending, receiving, executing, generating, calculating, storing,exposing, querying, processing, etc., in pursuance of the systems,apparatuses, and methods for displaying a tabular dataset and predictedvalues to a user display, as described herein. For example, hostorganization 110 of FIG. 1, machine 400 of FIG. 4, or system 1711 ofFIG. 17D may implement the described methodologies. Some of the blocksand/or operations listed below are optional in accordance with certainembodiments. The numbering of the blocks presented is for the sake ofclarity and is not intended to prescribe an order of operations in whichthe various blocks must occur.

At block 1791, processing logic receives a tabular dataset from a useras input, the tabular dataset having data values organized as columnsand rows.

At block 1792, processing logic identifies a plurality of null valueswithin the tabular dataset, the null values being dispersed acrossmultiple rows and multiple columns of the tabular dataset.

At block 1793, processing logic generates indices from the tabulardataset of columns and rows, the indices representing probabilisticrelationships between the rows and the columns of the tabular dataset.

At block 1794, processing logic displays the tabular dataset as outputto the user, the displayed output including the data values depicted asknown values and the null values depicted as unknown values.

At block 1795, processing logic receives input from the user to populateat least a portion of the unknown values within the displayed tabulardataset with predicted values.

At block 1796, processing logic queries the indices for the predictedvalues.

At block 1797, processing logic displays the predicted values as updatedoutput to the user.

Notably, blank values represented by the “unknown values” or “nullvalues” within a dataset may occur anywhere within a tabular dataset andyet permit a user to intuitively explore the dataset by having theanalysis engine's core analyze and seamlessly enable users to fill invalues wherever data is missing according to various criteria, such asminimum confidence thresholds, a user configurable slider mechanism suchas the one presented via the GUIs of FIGS. 17A-C, and so forth.Microsoft's Excel program is very good at calculating the next columnover or the next row down based on an algorithm, but such conventionalspreadsheet programs cannot tolerate missing values or holes within thedataset across different rows and different columns, especially whenthere are multiple unknown values within a single row or multiplemissing values within a single column.

The tabular dataset analyzed and displayed back to the user does notoperate by copying a known algorithm to another cell location based on arelational position in the manner that an Excel spreadsheet may operate.Rather, the population and display of missing or unknown values to auser is based on querying for and receiving predicted values for therespective cell location which is then displayed to the user within thetabular dataset displayed back to the user. This is made possiblethrough the analysis and generation of probabilistic based indices fromthe originally received dataset. Conventional solutions such as Excelspreadsheets simply do not perform such analysis nor do they generatesuch indices, and as such, they cannot render predicted values formultiple missing cells spread across multiple rows and columns of thedataset.

According to one embodiment, predictions for all of the missing cellsare determined for an entire tabular dataset received, and then as theuser selects a particular certainty level (e.g., such as a minimumconfidence level, etc.) the display is then updated with the values thatmeet the criteria. For instance, cells with missing values having apredicted value with a corresponding confidence indicator in excess of adefault threshold or a user specified threshold may then be displayed tothe user.

According to another embodiment of method 1721, generating indices fromthe tabular dataset of columns and rows further includes storing theindices within a database of the host organization; and in whichquerying the indices for the predicted values includes querying thedatabase for the predicted values.

According to another embodiment of method 1721, receiving input from theuser to populate at least a portion of the unknown values within thedisplayed tabular dataset with predicted values includes receiving inputfrom the user to populate all unknown values within the displayedtabular dataset with predicted values; in which querying the indices forthe predicted values includes querying the indices for a predicted valuefor every null value identified within the tabular dataset; and in whichdisplaying the predicted values as updated output to the user includesreplacing all unknown values by displaying corresponding predictedvalues.

According to another embodiment of method 1721, the plurality of nullvalues within the tabular dataset are not restricted to any row orcolumn of the tabular dataset; and in which displaying the predictedvalues as updated output to the user replaces the unknown valuesdisplayed with the tabular dataset without restriction to any row orcolumn and without changing the indices upon which the predicted valuesare based.

According to another embodiment of method 1721, querying the indices forthe predicted values includes querying the indices for each and everyone of the identified plurality of null values within the tabulardataset; in which the method further includes receiving the predictedvalues for each and every one of the identified plurality of null valueswithin the tabular dataset responsive to the querying; and in whichdisplaying the predicted values as updated output to the user includesdisplaying the received predicted values.

According to another embodiment of method 1721, querying the indices forthe predicted values includes: generating a Predictive Query Language(PreQL) query specifying a PREDICT command term for each and every oneof the identified plurality of null values within the tabular dataset;issuing each of the generated PreQL queries to a Predictive QueryLanguage Application Programming Interface (PreQL API); and receiving apredicted result for each and every one of the identified plurality ofnull values within the tabular dataset responsive to the issued PreQLqueries.

According to another embodiment of method 1721, displaying the tabulardataset further includes: displaying the known values using black textwithin cells of a spreadsheet; displaying the unknown values as blankcells within the spreadsheet; and displaying the predicted values usingcolored or grayscale text within the cells of the spreadsheet.

According to another embodiment of method 1721, displaying the predictedvalues as updated output to the user includes displaying the updatedoutput within a spreadsheet or table at a Graphical User Interface(GUI); in which the known values are displayed as populated cells withinthe spreadsheet or table at the GUI in a first type of text; in whichpredicted values are displayed as populated cells within the spreadsheetor table at the GUI in a second type of text discernable from the firsttype of text corresponding to the known values; and in which anyremaining unknown values are displayed as empty cells within thespreadsheet or table at the GUI.

According to another embodiment of method 1721, displaying the tabulardataset as output to the user and displaying the predicted values asupdated output to the user includes displaying the tabular dataset andthe predicted values within a spreadsheet or table at a Graphical UserInterface (GUI); in which the GUI further includes a slider interfacecontrollable by the user to specify an acceptable degree of uncertaintyfor the spreadsheet or table; and in which receiving input from the userto populate at least a portion of the unknown values within thedisplayed tabular dataset with predicted values includes receiving theacceptable degree of uncertainty as input from the user via the sliderinterface.

According to another embodiment, method 1721 further includes:displaying a minimum fill percentage for the GUI, wherein the minimumfill percentage corresponds to a percentage of known values within thetabular dataset from a sum of all null values and all known values forthe tabular dataset.

According to another embodiment of method 1721, the slider interfacecontrollable by the user to specify the acceptable degree of uncertaintyfor the spreadsheet or table is restricted to a range encompassing theminimum fill percentage and a maximum degree of uncertainty necessary tocompletely populate the displayed tabular dataset.

For instance, completely populating the displayed tabular dataset willresult in a 100% fill percentage but does not necessarily require theuser to specify an acceptable degree of uncertainty equal to 100%.Rather, it may be feasible that the displayed tabular dataset attains a100% fill percentage (e.g., every single null or unknown value ispopulated with a predictive result) at a user specified acceptabledegree of uncertainty of, by way of example, 50%. Regardless of whichacceptable degree of uncertainty the user specifies, as the acceptabledegree of uncertainty is increased, a greater portion of the table willbe filled, and as the acceptable degree of uncertainty is decreased, alesser portion of the table will be filled, thus permitting the user todynamically explore how predictive confidence affects the displayedresults in a highly intuitive manner.

According to another embodiment, method 1721 further includes:populating the spreadsheet or table of the GUI to a 100% fill percentageresponsive to input by the user specifying a maximum acceptable degreeof uncertainty via the slider interface; and populating all null valuesby degrading a required confidence for each of the predicted valuesuntil a predicted value is available for every one of the plurality ofnull values within the tabular dataset.

Unknown values correspond to data that is simply missing from thetabular dataset, whereas known values may be defined as those valuesthat are truly certain because the data was actually observed. Thus, aninitial presentment of the tabular dataset back to the user as outputmay include all values that are truly certain, that is, the initialoutput may simply display back all values actually observed within theoriginal tabular dataset in a table or spreadsheet type format. Unknownvalues will therefore still be missing. However, such a display may bedisplayed at 100% confidence because only known data is displayed. Thislevel of fill or this extent of population for the displayed outputtherefore corresponds to the minimum fill percentage, a value which mayalso be displayed to the user.

At the opposite end of the spectrum, the user may request to see a fullypopulated table, despite the originally presented tabular dataset havingunknown values. This may be accomplished by presenting the users allpredicted values having greater than zero certainty, and thus defined asfully filling in the displayed table or fully populating the displayedtable. When fully filling the table, any blanks identified will beprovided with a predicted value for display regardless of confidence forthe predicted value. Thus, all values between a “0” certainty and “1”certainty are displayed. Such a view is available to the user, however,the display may additionally indicate that certainty for certainpredicted values is poor or indicate a confidence score for thepredicted value with a lowest confidence quality, and so forth. Inalternative embodiments, a user may specify a minimum confidence qualitythreshold and then displayed values will be restricted on the basis ofthe user specified minimum confidence quality threshold. Where a userspecified minimum confidence quality threshold is specified as beinggreater than zero the maximum fill percentage may fall below 100% asthere are likely cells that are not capable of being predicted with aconfidence in excess of the user specified minimum confidence qualitythreshold.

Thus, in accordance with another embodiment, method 1721 furtherincludes: displaying a user controllable minimum confidence threshold ata Graphical User Interface (GUI) displaying the tabular dataset asoutput to the user within a spreadsheet or table; receiving a userspecified minimum confidence threshold as input via the usercontrollable minimum confidence threshold; and in which displaying thepredicted values as updated output to the user includes displaying onlythe predicted values at the GUI having a corresponding confidenceindicator equal to or greater than the user specified minimum confidencethreshold.

In certain embodiments, queries are constructed and then issued forevery missing cell or unknown value within the tabular dataset andpredictions are then responsively returned. Taking one of those missingvalues, a confidence indicator may be returned as a value or adistribution may be returned which allows for further analysis. Take forexample, a particular missing cell which is then used to query for apredicted true|false value. The query may return the results of anexemplary 100 predictions. Perhaps 75 of the predictions return truewhereas 25 of the predictions return false. It may therefore be saidthat the value being predicted has a 75% certainty of being true. That75% certainty may then be compared against a threshold to determinewhether or not to display the value. There are, however, many other waysof computing a certainty or a confidence indicator besides this basicexample. In a more complex example, say the results of a prediction fora true|false value were 50-50, with the prediction results coming backas 50 true and 50 false. In such a case, although the result is 50%certain to be true and 50% certain to be false, the middle of the road50-50 result is also maximally uncertain. In other words, the 50-50result is the least certain result possible, and thus, corresponds tomaximal uncertainty.

Predictions are not limited to simply true|false. Take for example anull value for an RGB field in which there is a closed set with threecolor possibilities; red, green and blue. Here the prediction may return100 exemplary guesses or predictions, as before, but now attempting topredict the color value as one of red, green, or blue. Thus, the resultsmay have a small percentage of the results as red, a much largerpercentage as green, and some medium percentage as blue. With such aresult, the predicted value for the unknown cell may therefore bereturned as green with the certainty being the proportion of attemptedpredictions that returned green out of all guesses. Thus, if 100attempts were made to determine the RGB value and 43 of those came backgreen, then it may be determined that certainty is 43 percent to begreen. Again, many other examples and interpretations of a returneddistribution are feasible. In certain situations the analysis enginesimply returns a value or score representative of confidence orcertainty in the result whereas in other situations, distributions arereturned representative of many attempts made to render the predictedvalue.

According to another embodiment, method 1721 further includes:displaying a user controllable minimum confidence threshold at aGraphical User Interface (GUI) displaying the tabular dataset as outputto the user within a spreadsheet or table; and displaying a maximum fillpercentage for the GUI, in which the maximum fill percentage correspondsto a sum of all known values and all null values returning a predictedvalue with a confidence indicator in excess of the user controllableminimum confidence threshold as a percentage of a sum of all null valuesand all known values.

According to another embodiment, method 1721 further includes: receivinga confidence indicator for every one of the plurality of null valueswithin the tabular dataset responsive to querying the indices for thepredicted values; and in which displaying the predicted values asupdated output to the user includes displaying selected ones of thepredicted values that correspond to a confidence quality in excess of adefault minimum confidence threshold or a user specified minimumconfidence threshold when present.

According to one embodiment, queries are issued for every unknown valueresponsive to which predicted values are returned and then ranked orordered according to their corresponding confidence indicators. When theslider is at 100 percent fill per the user input the display is updatedto show all cells with either known values or predicted valuesregardless of the confidence for the predicted values. Conversely, ifthe user's minimum threshold input field is set to 100% then only theknown values will be displayed. Dropping the certainty threshold to 75%will then render display output having all known values (which are bynature 100% certain) along with any predicted value having a certaintyindicator of at least 75%, and so forth. In such a way, the user mayintuitively manipulate the controls to explore and interact with thedata.

According to another embodiment, method 1721 further includes: receivinga distribution for every one of the plurality of null values within thetabular dataset responsive to querying the indices for the predictedvalues; calculating a credible interval for each distribution received;and in which displaying the predicted values as updated output to theuser includes displaying selected ones of the predicted values thatcorrespond to a calculated credible interval in excess of a minimumthreshold.

A credible interval (or a Bayesian confidence interval) is an intervalin the domain of a posterior probability distribution used for intervalestimation. The generalization to multivariate problems is the credibleregion. For example, in an experiment that determines the uncertaintydistribution of parameter t, if the probability that t lies between 35and 45 is 0.95, then 35<=t<=45 is a 95% credible interval.

According to another embodiment of method 1721, displaying the tabulardataset further includes: displaying the known values as a first texttype within cells of a spreadsheet; querying the indices for a predictedvalue corresponding to every one of the unknown values; and in whichdisplaying the predicted values as updated output to the user includesdisplaying the predicted values as a second text type within the cellsof the spreadsheet, in which the second text type has a displayedopacity in proportion to a confidence indicator for the predicted valuedisplayed.

According to another embodiment of method 1721, displaying the tabulardataset as output to the user includes displaying the known values asblack text within cells of a spreadsheet; and in which displaying thepredicted values as updated output to the user includes displaying thepredicted values as grayscale text with the predicted values having ahigher confidence indicator being displayed at darker grayscales thanthe predicted values having a lower confidence indicator.

For example, in place of using a slider, all values may be provided tothe user as display output. For instance, known values may be depictedin pure-black text and then predicted values may be distinguished bydisplaying them as grayscale text with their intensity or their opacitybeing proportional to their certainty. In such a way, a predicted valuehaving high confidence may be displayed in dark gray but not quite blacktext and conversely, a predicted value having low confidence may stillbe displayed, but in light gray text.

According to another embodiment, method 1721 further includes:displaying a prediction difficulty score for every column of the tabulardataset displayed as output to the user on a per-column basis, in whichthe prediction difficulty score is calculated for each column of thetabular dataset by: (i) identifying all unknown values within thecolumn; (ii) querying the indices for a predicted value corresponding toeach of the unknown values identified within the column; (iii) receivinga confidence indicator for each of the unknown values identified withinthe column; and (iv) calculating the prediction difficulty score for thecolumn based on the confidence indicators received for the unknownvalues identified within the column.

According to another embodiment of method 1721, the method furtherincludes: displaying a maximum fill percentage for every column of thetabular dataset displayed as output to the user on a per-column basis,in which the maximum fill percentage is based on a quantity of theunknown values identified within the respective column having confidenceindicators exceeding a minimum confidence quality threshold.

For example, taking each column in the original tabular dataset therewill be an indication to the user regarding how much of the particularcolumn may be populated using predicted values or a combination of knownand predicted values while conforming to a minimum confidence threshold.Thus, to attain a 100% fill for a particular column it may necessary tolower the minimum confidence drastically. As certainty is decreased moreof each column is capable of being filled by replacing unknown valueswith predicted values. Certain columns are likely to be more easilypredicted and thus, they may reach 100% fill for a given certainty whileother columns at the same certainty will remain partially unfilled.Regardless, such a display to the user enables simple and intuitiveexploration of the data by the user with a minimal learning curve andwithout a deep technical understanding of the probability techniquescausing the predicted data values to be rendered.

According to one embodiment there is a non-transitory computer readablestorage medium having instructions stored thereon that, when executed bya processor in a host organization, the instructions cause the hostorganization to perform operations including: receiving a tabulardataset from a user as input, the tabular dataset having data valuesorganized as columns and rows; identifying a plurality of null valueswithin the tabular dataset, the null values being dispersed acrossmultiple rows and multiple columns of the tabular dataset; generatingindices from the tabular dataset of columns and rows, the indicesrepresenting probabilistic relationships between the rows and thecolumns of the tabular dataset; displaying the tabular dataset as outputto the user, the displayed output including the data values depicted asknown values and the null values depicted as unknown values; receivinginput from the user to populate at least a portion of the unknown valueswithin the displayed tabular dataset with predicted values; querying theindices for the predicted values; and displaying the predicted values asupdated output to the user.

FIG. 18 depicts feature moves 1801 and entity moves 1802 within indicesgenerated from analysis of tabular datasets. On the left a feature move1801 is depicted among the three views shown: view 1 at element 1810,view 2 at element 1811, and view 3 at element 1812. As depicted, feature1805 (e.g., such as a column, characteristic, etc.) can be moved eitherto another existing 1820 view as is done via the arrow pointing left tomove feature 1805 from view 2 at element 1811 to view 1 at element 1810,or alternatively, the feature 1805 may be moved to a new view 1821, asis happening to feature 1805 as depicted by the right facing arrow tomove feature 1805 to the new view 4 at element 1813.

On the right an entity move 1802 is depicted among the two categoriesshown: category 1 at element 1825 and category 2 at element 1826. Asdepicted, a entity 1806 (e.g., such as a row) can be moved either toanother existing 1823 category as is depicted by the arrow pointing downto move entity 1806 from category 1 at element 1825 to category 2 atelement 1826, or alternatively, the entity 1806 may be moved to a newcategory 1824, as is happening to entity 1806 as depicted by the longerdownward facing arrow to move entity 1806 to the new category 3 atelement 1827.

FIG. 19A depicts a specialized GUI 1901 to query using historical dates.The specialized GUI 1901 implementation depicted here enables users tofilter on a historical value by comparing a historical value versus acurrent value in a multi-tenant database system. Filtering forhistorical data is enabled via the GUI's “Close date (Historical)” dropdown box or similar means (e.g., a calendar selector, etc.) in which theGUI 1901 displays current fields related to historical fields.

The GUI 1901 enables users to filter historical data by comparing ahistorical value versus a constant in a multi-tenant database system.The GUI 1901 utilizes the analysis engine's predictive capabilities byconstructing and issuing the appropriate API calls on behalf of the userwithout requiring users of the GUI to understand how the API calls areconstructed or even which command terms or parameters need to bespecified to yield the appropriate output, and in such a way, the GUI1901 provides a highly intuitive interface for users without a steeplearning curve.

The GUI 1901 executes the necessary queries or API calls and thenconsumes the data which is then presented back to the end users via adisplay interface for the GUI 1901, such as a client device 106A-C asillustrated in FIG. 1. Consider for example, a salesperson looking atthe sales information in a particular data set. The GUI 1901 interfacecan take the distributions provided by the analysis engine and produce avisual indication for ranking the information according to a variety ofcustomized solutions and use cases.

For example, SalesCloud is an industry leading CRM application that iscurrently used by 135,000 enterprise customers. Such customersunderstand the value of storing their data in the Cloud and appreciate aweb based GUI 1901 interface to view and act on their data. Suchcustomers frequently utilize report and dashboard mechanisms provided bythe cloud based service. Presenting these various GUIs as tabbedfunctionality enables salespeople and other end users to explore theirunderlying dataset in a variety of ways to learn how their business isperforming in real-time. These users may also rely upon partners toextend the provided cloud based service capabilities through additionalGUIs that make use of the APIs and interfaces that are described herein.

A cloud based service that offers customers the opportunity to learnfrom the past and draw data driven insights is highly desirable as suchfunctionality may help these customers make intelligent decisions aboutthe future for their business based on their existing dataset. GUI 1901provides such an interface.

The customized GUIs utilize the analysis engine's predictivefunctionality to implement reports which rely upon predictive resultswhich may vary per customer organization or be tailored to a particularorganizations needs via programmatic parameters and settings exposed tothe customer organization to alter the configuration and operation ofthe GUIs and the manner in which they execute API calls against theanalysis engine's functionality.

For instance, a GUI 1901 may be provided to compute and assign anopportunity score based on probability for a given opportunityreflecting the likelihood of that opportunity to close as a win or loss.The data set to compute this score consists of all the opportunitiesthat have been closed (either won/loss) in a given period of time, suchas 1, 2, or 3 years or a lifetime of an organization, etc., and such aduration may be configured using the date range controls of GUI 1901 tospecify the date range, even if that range is in the past.

Additional data elements from the customer organization's dataset mayalso be utilized, such as an accounts table as an input. Machinelearning techniques implemented via the analysis engine's core, such asSVN, Regression, Decision Trees, PGM, etc., are then used to build anappropriate model to render the opportunity score and then the GUI 1901depicts the information to the end user via the interface.

FIG. 19B depicts an additional view of a specialized GUI 1902 to queryusing historical dates. The specialized GUI 1902 implementation depictedhere enables users to determine the likelihood of an opportunity toclose using historical trending data. For instance, GUI 1902 permitsusers to easily configure the predictive queries using the “history”selector for picking relative or absolute dates.

With this GUI 1902 users are enabled to look at how an opportunity haschanged over time, independent of stage, etc. The user can additionallylook at how that opportunity has matured from when it was created untilwhen it was closed. For instance, GUI 1902 has set a historical data ofJan. 1, 2013 through Mar. 1, 2013 using the date configuration controlsand the table at the bottom depicts that the amount of the opportunityhas decreased by $10,000 but the stage was and still is in a“prospecting” phase.

The GUI 1902 additionally enables users to determine the likelihood ofan opportunity to close at a given stage using historical trending data.Where GUI 1901 above operates independent of stage of the salesopportunity, GUI 1902 focuses on the probability of closing at a givenstage as a further limiting condition for the closure. Thus, customersare enabled to use the historical trending data to know exactly when thestage has changed and then additionally predict what factors wereinvolved to move from stage 1 to 2, from stage 2 to 3 and so forth.

The GUIs additionally permit the users to predict an opportunity toclose on the basis of additional social and marketing data provided atthe interface or specified by the user. For example, the dataset of thecustomer organization or whomever is utilizing the system may beexpanded on behalf of the end user beyond the underlying dataset byincorporating such social and marketing data which is then utilized bythe analysis engine to further influence and educate the predictivemodels. For instance, certain embodiments pull information from anexemplary website such as “data.com,” and then the data is associatedwith each opportunity in the original dataset by the analysis enginewhere feasible to discover further relationships, causations, and hiddenstructure which can then be presented to the end user. Other datasources are equally feasible, such as pulling data from socialnetworking sites, search engines, data aggregation service providers,etc.

In one embodiment, social data is retrieved and a sentiment is providedto the end-user via the GUI to depict how the given product is viewed byothers in a social context. Thus, a salesperson can look at a customer'sLinkedIn in profile and with information from data.com or other sourcesthe salesperson can additionally be given sentiment analysis in terms ofsocial context for the person that the salesperson is actually trying tosell to. For instance, such data may reveal whether the target purchaserhas commented about other products or perhaps has complained about otherproducts, etc. Each of these data points and others may help influencethe model employed by the analysis engine to further improve a renderedprediction.

In another embodiment, determining the likelihood for an opportunity toclose is based further on industry specific data retrieved from sourcesexternal to an initially specified dataset. For instance, rather thanusing socially relevant data for social context of sentiment analysis,industry specific data can be retrieved and input to the predictivedatabase upon which analysis engine performs its modeling as describedabove, and from which further exploration can then be conducted by usersof the dataset now having the industry specific data integrated therein.

According to other embodiments, datasets are explored beyond theboundaries of any particular customer organization having data withinthe multi-tenant database system. For instance, in certain embodiments,benchmark predictive scores are generated based on industry specificlearning using cross-organizational data stored within the multi-tenantdatabase system. For example, data mining may be performed againsttelecom specific customer datasets, given their authorization or licenseto do so. Such cross-organization data to render a much largermulti-tenant dataset can then be analyzed via the analysis engine'smodels and provide insights, relationships, causations, and additionalhidden structure that may not be present within a single customerorganizations' dataset. For instance, if a customer is trying to close a$100 k deal in the NY-NJ-Virginia tri-city area, the probability forthat deal to close in 3 months may be 50%, according to such analysis,because past transactions have shown that it takes up to six months toclose a $100 k telecom deal in NY-NJ-Virginia tri-city area when viewedin the context of multiple customer organizations' datasets. Many of theinsights realized through such a process may be non-intuitive, yetcapable of realization through application of the techniques describedherein.

With industry specific data present within a given dataset it ispossible to delve even deeper into the data and identify benchmarksusing such data for a variety of varying domains across multipledifferent industries. For instance, based on such data predictiveanalysis may review that, in a given region it takes six months to sellsugar in the Midwest and it takes three months to sell laptops in theEast Coast, and so forth.

Then, if a new opportunity arises and a vendor is trying to, forexample, sell watches in California, the vendor can utilize suchinformation to gain a better understanding of the particular regionalmarket based on the predictions and confidence levels given.

FIG. 19C depicts another view of a specialized GUI 1903 to configurepredictive queries. The analysis engine's predictive functionality canadditionally reveal information for a vertical sector as well as for theregion. When mining a customer organization's dataset a relationship maybe discovered that, where customers bought items “a,” those customersalso bought item “b.” These kinds of matching relationships are useful,but can be further enhanced. For instance, using the predictive analysisof the analysis engine it is additionally possible to identify the setof factors that led to a particular opportunity score.

As depicted here, the GUI 1903 presents a 42% opportunity at the userinterface but when the user cursors over (e.g., a mouse over event,etc.) the opportunity score, the GUI 1903 then displays sub-detailhaving additional elements that make up that opportunity score. The GUI1903 again constructs and issues the necessary API calls on behalf ofthe user such that an appropriate predictive command term is selectedand executed against the indices to pull the opportunity score andrelevant display information to the user including the sub-detailrelationships and causations considered relevant.

The GUI 1903 can additionally leverage the PREDICT and ANALYZE commandterms by triggering the appropriate function for a given opportunity asspecified by the user at the GUI 1903 to return the raw data needed bythe GUI 1903 to create a histogram for the opportunity. Thus, not onlycan the user be given a score, but the user may additionally be giventhe relevant factors and guidance on how to interpret the information soas to assist the user with determining an appropriate call to actiongiven the information provided.

Moreover, as the end-users, such as salespersons, see the data and actupon it, a feedback loop is created through which further data is inputinto the predictive database upon which additional predictions andanalysis are carried out in an adaptive manner. For example, as theanalysis engine learns more about the dataset associated with theexemplary user or salesperson, the underlying models may be refreshed ona recurring basis by re-performing the analysis of the dataset so as tore-calibrate the data using the new data obtained via the feedback loop.Such new data may describe whether sales opportunities closed with asale or loss, identify the final amount, timing, resources involved, andso forth, all of which help to better inform the models and in turnrender better predictions for other queries going forward.

FIG. 19D depicts an exemplary architecture in accordance with describedembodiments. In particular, customer organizations 1905A, 1905B, and1905C are depicted, each with a user's client device and display 1906A,1906B, and 1906C capable of interfacing with host organization 1910 vianetwork 1925, including sending input, queries, and requests andresponsively receiving responses including output for display. Withinhost organization 1910 is a request interface 1976 which may optionallybe implemented by web-server 1975. The host organization furtherincludes processor(s) 1981, memory 1982, a query interface 1980,analysis engine 1985, and a multi-tenant database system 1930. Withinthe multi-tenant database system 1930 are execution hardware, software,and logic 1920 that are shared across multiple tenants of themulti-tenant database system 1930, authenticator 1998, and a predictivedatabase 1950 capable of storing indices generated by the analysisengine 1985 to facilitate the return of predictive record setsresponsive to queries executed against the predictive database 1950.

According to one embodiment, the host organization 1910 operates asystem 1911 having at least a processor 1981 and a memory 1982 therein,in which the system 1911 includes a request interface 1976 to receiveinput from a user device 1906A-C specifying a dataset 1953 of sales datafor a customer organization 1905A-C, in which the sales data specifies aplurality of sales opportunities; an analysis engine 1985 to generateindices 1954 from rows and columns of the dataset 1953, the indicesrepresenting probabilistic relationships between the rows and thecolumns of the dataset 1953; a predictive database 1950 to store theindices 1954; the analysis engine 1985 to select one or more of theplurality of sales opportunities specified within the sales data; aquery interface 1980 to query 1957 the indices 1954 for a win or losepredictive result 1958 for each of the selected one or more salesopportunities; and in which the request interface 1976 is to furtherreturn the win or lose predictive result 1958 for each of the selectedone or more sales opportunities as display output 1955 to the userdevice 1906A-C.

The request interface 1976 may additionally receive user event input1956 from a user device 1906A-C indicating one of the displayed one ormore sales opportunities or their corresponding win or lose predictiveresult 1958, responsive to which the user interface may provideadditional drill-down sub-detail. For instance, if a user of atouchscreen touches one of the displayed opportunities or clicks on oneof them then the user interface may communicate such input to therequest interface 1976 causing the host organization to provide updateddisplay output 1959 with additional detail for the specified salesopportunity, such as relevant characteristics, etc.

According to another embodiment, the system 1911 further includes apredictive database 1950 to store the indices generated by the analysisengine. In such an embodiment, the predictive database 1950 is toexecute as an on-demand cloud based service at the host organization1910 for one or more subscribers.

In another embodiment, the system 1911 further includes an authenticator1998 to verify the user (e.g., a user at one of the user's client deviceand display 1906A-C) as a known subscriber. The authenticator 1998 thenfurther operates to verify authentication credentials presented by theknown subscriber.

In another embodiment, the system 1911 further includes a web-server1975 to implement the request interface; in which the web-server 1975 isto receive as input, a plurality of access requests from one or moreclient devices from among a plurality of customer organizationscommunicably interfaced with the host organization via a network; amulti-tenant database system with predictive database functionality toimplement the predictive database; and in which each customerorganization is an entity selected from the group consisting of: aseparate and distinct remote organization, an organizational groupwithin the host organization, a business partner of the hostorganization, or a customer organization that subscribes to cloudcomputing services provided by the host organization.

FIG. 19E is a flow diagram illustrating a method in accordance withdisclosed embodiments. Method 1921 may be performed by processing logicthat may include hardware (e.g., circuitry, dedicated logic,programmable logic, microcode, etc.), software (e.g., instructions runon a processing device to perform various operations such transmitting,sending, receiving, executing, generating, calculating, storing,exposing, querying, processing, etc., in pursuance of the systems,apparatuses, and methods for rendering scored opportunities using apredictive query interface, as described herein. For example, hostorganization 110 of FIG. 1, machine 400 of FIG. 4, or system 1911 ofFIG. 19D may implement the described methodologies. Some of the blocksand/or operations listed below are optional in accordance with certainembodiments. The numbering of the blocks presented is for the sake ofclarity and is not intended to prescribe an order of operations in whichthe various blocks must occur.

At block 1991, processing logic receives input from a user devicespecifying a dataset of sales data for a customer organization, in whichthe sales data specifies a plurality of sales opportunities.

At block 1992, processing logic generates indices from rows and columnsof the dataset, the indices representing probabilistic relationshipsbetween the rows and the columns of the dataset.

At block 1993, processing logic stores the indices in a queryabledatabase within the host organization.

At block 1994, processing logic selects one or more of the plurality ofsales opportunities specified within the sales data.

At block 1995, processing logic queries the indices for a win or losepredictive result for each of the selected one or more salesopportunities.

At block 1996, processing logic displays the win or lose predictiveresult for each of the selected one or more sales opportunities to theuser device as output.

The User Interface (UI) or Graphical User Interface (GUI) consumes dataand predictive results returned from the predictive interface to displaythe predicted results to the user in a highly intuitive fashion alongwith other data such as scored sales opportunities, the quality ofpredictions, and what factors or characteristics are probabilisticallyrelevant to the sales opportunities and other metrics displayed.

According to another embodiment of method 1921, querying the indices fora win or lose predictive result for each of the selected one or moresales opportunities includes: generating a Predictive Query Language(PreQL) query specifying a PREDICT command term for each of the selectedone or more sales opportunities; issuing each of the generated PreQLqueries to a Predictive Query Language Application Programming Interface(PreQL API); and receiving the win or lose predictive result for each ofthe selected one or more sales opportunities responsive to the issuedPreQL queries.

According to another embodiment of method 1921, the dataset of salesdata includes closed sales opportunities for which a win or lose resultis known and recorded within the dataset of sales data for each closedsales opportunity; in which the dataset of sales data further includesopen sales opportunities for which a win or lose result is unknown andcorresponds to a null value within the dataset of sales data for eachopen sales opportunity; and in which each of the plurality of selectedsales opportunities are selected from the open sales opportunities.

According to another embodiment of method 1921, querying the indices fora win or lose predictive result for each of the selected one or moresales opportunities includes: constructing a query specifying theselected sales opportunity, in which the query specifies a PREDICTcommand term and includes operands for the PREDICT command termincluding at least a row corresponding to the selected sales opportunityand a column corresponding to the win or lose result; and receiving thewin or lose predictive result for the row corresponding to the selectedsales opportunity responsive to issuing the constructed query.

According to another embodiment, method 1921 further includes: queryingthe indices for a predicted sales amount for each of the selected one ormore sales opportunities; and displaying the predicted sales amount withthe win or lose predictive result for each of the selected one or moresales opportunities to the user device as output.

According to another embodiment of method 1921, querying the indices fora predicted sales amount includes: constructing a query for each of theselected sales opportunities, in which each query specifies a PREDICTcommand term and includes operands for the PREDICT command termincluding at least a row corresponding to the selected sales opportunityand a column corresponding to a sales amount; and receiving thepredicted sales amount result for the row corresponding to the selectedsales opportunity responsive to issuing the constructed query.

Querying the predictive query interface returns the predictive resultbeing sought (e.g., win or lose prediction, predicted close amount,predicted close date, etc.), but additionally relevant is the quality ofthat prediction, that is to say, the probability or likelihood that arendered prediction will come true. The predictive query interface mayreturn a distribution, an interval, or other value depending on theconfiguration and the structure of the query issued. For instance, aconfidence quality indicator may be returned indicating a value betweenzero and a hundred, providing a quantitative metric by which to assessthe quality of the prediction.

Providing predicted win or lose results for existing sales opportunitiesalong with a measure of quality for the prediction rendered is helpfulto a salesperson who must evaluate which sales opportunities to target.Naturally, the salesperson is incentivized to spend time and resourceson the opportunities that are more likely to result in success. Thus,the output may aid a salesperson in evaluating which sales opportunitiesare likely to close and therefore which may be worked in an effort tomake sales quota and maximize their commissions.

According to another embodiment, method 1921 further includes: receivinga confidence indicator for each of the win or lose predictive results;and displaying the confidence indicator with the output to the userdevice with each of the win or lose predictive results displayed for theone or more sales opportunities selected.

According to another embodiment, method 1921 further includes: receivinga confidence indicator for each of the selected one or more salesopportunities with the win or lose predictive results responsive to thequerying; and displaying the confidence indicators received as output tothe user device concurrently with displaying the win or lose predictiveresult for each of the selected one or more sales opportunities.

Certain embodiments may utilize a benchmark or threshold, such as 70% orsome other default or user configured value to establish the minimumconfidence quality required for sales opportunities to be returned tothe user display. A second threshold may be required for thoseembodiments which further display a recommendation to the user display.

According to another embodiment of method 1921, selecting one or more ofthe plurality of sales opportunities includes selecting all salesopportunities in a pre-close sales stage and having an unknown win orlose result; identifying the one or more of the plurality of salesopportunities having a win or lose predictive result in excess of aminimum confidence indicator threshold; and in which displaying the winor lose predictive result to the user device as output includesdisplaying the one or more of the plurality of sales opportunitiesidentified as having the win or lose predictive result in excess of theminimum confidence indicator threshold.

For any given sales opportunity there may be multiple sales stages and asales opportunity may be in an open state or a closed stage (e.g., anopen state may be a pre-close state or any stage prior to closure of theopportunity). The sales opportunity may also be in any of a number ofinterim stages, especially for sales teams that deal with largecustomers and handle large sales transactions. Such stages make up thesales life cycle. For example, there may be a discovery stage in which asalesperson works to determine what the right product is for a givencustomer, a pricing or quote stage where pricing is determined anddiscounts are negotiated, and so forth.

For large transactions, the sales life cycle may last three months, sixmonths, sometimes nine months, largely depending on the size andcomplexity of a sales opportunity. Very complex transactions can spanmany years, for instance, where a customer is considering amulti-billion dollar commitment, such as for aircraft engines. Smallerless complex transactions, such as a contract for database software, maymove along more quickly.

According to another embodiment, method 1921 further includes:displaying a recommendation as output to the user device, in which therecommendation specifies at least one of the plurality of salesopportunities in a pre-close sales stage; in which the recommendationspecifies for the output (i) the at least one of the plurality of salesopportunities in a pre-close sales stage by sales opportunity name asspecified by the dataset of sales data; and in which the recommendationfurther specifies for the output, one or more of: (ii) the win or losepredictive result indicating a sales win; (iii) a confidence indicatorfor the win or lose predictive result indicating the sales win; and (iv)a predicted sales amount; and (v) a predicted sales opportunity closedate.

Close dates may be predicted and provided as output which can be highlyrelevant for the purposes of sales forecasting. For instance, if asalesperson states to management that a deal will close in fiscal Q1 butthe prediction returns a high confidence sales close date of fiscal Q3,then it may be appropriate to either adjust the sales forecast or changestrategy for a given sales opportunity (e.g., increase urgency, improvepricing terms, discounts, etc.).

User Interfaces additionally provide means by which a user may changedefault values, specify relevant historical date ranges upon whichpredictions and queries are to be based, specify the scope of a datasetto be utilized in making predictions, and so forth.

For instance, a user administrative page equivalent to those describedabove at FIGS. 19A, 19B, and 19C provide reporting capabilities throughwhich a user may specify the input sources (e.g., the dataset of salesdata for a customer organization), restrictions, filters, historicaldata, and other relevant data sources such as social media data, updatedsales data, and so forth.

According to another embodiment of method 1921, the recommendation isdetermined based on weightings assigned to the output specified at (i)through (iv); and in which the weightings are assigned by defaults andare custom configurable via a Graphical User Interface (GUI) displayedat the user device.

According to another embodiment of method 1921, selecting one or more ofthe plurality of sales opportunities includes selecting salesopportunities in a closed sales stage and having a known win or loseresult; querying the indices for a win or lose predictive result foreach of the selected one or more sales opportunities in a closed salesstage and having the known win or lose result, in which the win or losepredictive result ignores the known win or lose result; determiningpredictive accuracy for each of the plurality of sales opportunitiesselected by comparing the known win or lose result against the win orlose predictive result; and displaying the determined predictiveaccuracy for each of the plurality of sales opportunities selected asoutput to the user device with the win or lose predictive resultdisplayed.

According to another embodiment, method 1921 further includes: receivingdate range input from a GUI displayed at the user device, the date rangeinput specifying a historical date range upon which the win or losepredictive result is based.

According to another embodiment of method 1921, receiving input from auser device specifying a dataset of sales data for a customerorganization includes at least one of: receiving the dataset as a tablehaving the columns and rows; receiving the dataset as data stream;receiving a spreadsheet document and extracting the dataset from thespreadsheet document; receiving the dataset as a binary file created bya database; receiving one or more queries to a database and responsivelyreceiving the dataset by executing the one or more queries against thedatabase and capturing a record set returned by the one or more queriesas the dataset; receiving a name of a table in a database and retrievingthe table from the database as the dataset; receiving search parametersfor a specified website and responsively querying the search parametersagainst the specified website and capturing search results as thedataset; and receiving a link and authentication credentials for aremote repository and responsively authenticating with the remoterepository and retrieving the dataset via the link.

According to another embodiment, method 1921 further includes: receivingentity selection input from a GUI displayed at the user device, theentity selection input specifying one of the win or lose predictiveresults displayed to the user device as output; displaying sub-detailfor one of the sales opportunities as updated output to the user deviceresponsive to the entity selection input; and in which the sub-detailincludes one or more features probabilistically related to the win orlose predictive results displayed.

According to another embodiment of method 1921, the entity selectioninput includes one of a mouse over event, a cursor over event, a clickevent, a touchscreen selection event, or a touchscreen position eventcorresponding to one of the win or lose predictive results displayed;and in which displaying sub-detail includes displaying the sub-detailwithin a graphical overlay positioned on top of and at least partiallycovering the win or lose predictive results displayed initially.

According to another embodiment, method 1921 further includes:constructing a query to retrieve the one or more featuresprobabilistically related to the win or lose predictive resultsdisplayed; in which the query includes a RELATED command term and atleast one operand for the RELATED command term specifying a columncorresponding to a win or lose result column.

For example, it may be determined that a sales opportunity sponsor turnsout to be probabilistically related to the win or lose of a sale. Thus,the UI additionally provides functionality to track available sponsors,such as satisfied customers or high level executives that can speak witha potential customer in an effort to improve the likelihood of successfor a given sales opportunity.

The User Interface may additionally construct and issue a SIMILARcommand term to the predictive database to return and display salesopportunities that are most like a particular sales opportunity beingevaluated by the salesperson. Such data may help the salesperson to drawadditional insights from other similar sales opportunities which did ordid not result in a successful win.

By understanding what factors affect a particular sales opportunity, asalesperson may focus specifically on influencing those factors in aneffort to increase the likelihood of a successful close for a particularsales opportunity. For instance, if a conversation between the customerand the company's CEO proves to be helpful in certain types oftransactions, then that may be a worthwhile resource expenditure.Conversely, if a given type of pricing structure turns out to befavorable for certain customer types, products, or industries, then thatmay be a worthy consideration to increase the likelihood of success fora given sales opportunity. Exploration of such characteristics may bedone through the user interface, including manipulating values as “whatif” scenarios and then updating the predictive results to the userdisplay based on the “what if” scenario parameters.

According to another embodiment of method 1921, displaying thesub-detail includes: displaying column names for the one or morefeatures probabilistically related to the win or lose predictive resultsdisplayed; and displaying data values from the dataset corresponding torow and column intersects for the column names and an entity rowcorresponding to the one sales opportunity for the sub-detail displayed.

Social media data is one type of auxiliary data. A variety of datasources may be specified to further enhance the predictive resultsincluding, for example: contacts, accounts, account phases, accounttask, account contact person, account sponsor or referral, and so forth.

Social media data is available from sources including Radian 6 and BuddyMedia offered by salesforce.com. Such sources provide aggregated andstructured data gleaned from social media sources such as Facebook,Twitter, LinkedIn, and so forth. Using these sources, it may be possibleto associate an individual, such as John Doe, with a particular salesopportunity and then enhance the indices with data that is associatedwith John Doe within the social networking space. For example, perhapsJohn Doe has tweeted about a competitors product or the products offeredby the salesperson. Or their may be a news feed which mentions theproduct or the company or the sales opportunity targeted, or there maybe customer reviews which are contextually relevant, and so forth. Suchdata points can be integrated by specifying the appropriate sources atthe UI which will in turn cause the analysis engine to performadditional analysis to update the indices and if such data points areprobabilistically relevant, then their relationship will affect thepredictive results and be discoverable through the user interface.

In certain embodiments, benchmarking capabilities are provided whichenable a user to analyze supplemental data sources based on, forexample, a manufacturing industry versus a high tech industry, or datawhich is arranged by customers in geographical region, and so forth.While such data is not typically maintained by a customer organization,it can be sourced and specified as additional supplemental data throughthe user interface upon which the analysis engine's core can update theindices and further improve the predictive results or yield furtherinsights into a customer organization's data that may not otherwise befeasible.

In one embodiment, such supplemental data sources are provided throughthe User Interface as part of an existing cloud based subscription orupon the payment of an additional fee. For example, a customer maypurchase a data package which enables them to integrate industrybenchmarking data to perform analysis for the customer organization'sspecific sales opportunities in view of aggregated benchmarking data fora collection of potential sales customers or for a collection ofpotential verticals, and so forth. Once this additional data isspecified and analyzed and the indices are updated, the user may thenexplore the data and its affect upon their predictive results throughthe UI provided.

In one embodiment, historical data is tracked and the scope ofhistorical data that may be analyzed, viewed, and otherwise explored bya user is based on subscription terms. For instance, a cloud basedservice subscriber may expose the relevant user interface to allcustomers for free, but then limit the scope of data that may beanalyzed to only an exemplary three months, whereas paying subscribersget a much deeper and fuller dataset, perhaps two years worth ofhistorical analysis. Certain embodiments operate on a dataset specifiedby the customer explicitly, whereas other embodiments may default to aparticular dataset on behalf of the customer based on the system'sknowledge of that customer's data already stored at a host organization.Notably, conventional databases do not track and expose a historicalview of data stored in a database. Change logs and roll backs areenabled on most databases, but conventional databases do not expose suchdata to queries because they are not intended for such a purpose.Conversely, the user interface described here permits the user tospecify a historical date range which then enables the user to explorehow data has changed over time or query the database in the perspectiveof a past date, resulting in query results returning the data as theywere at the past date, rather than as they exist in the present. Themethodologies described herein use a separate object to that databaseupdates may be fully committed and further so that change and audit logsmay be flushed without losing the historical data.

According to another embodiment, method 1921 further includes: receivingadditional input from the user device specifying a social media datasource; updating the indices based on the social media data sourcespecified; and displaying updated win or lose predictive results foreach of the selected one or more sales opportunities to the user deviceas output with characteristics derived from the social media data sourcedetermined to be relevant to the updated win or lose predictive results.

According to another embodiment of method 1921, the social media datasource corresponds to a social media data aggregator which listens tosocial media networks and provides aggregated social media data asstructured output.

According to another embodiment, method 1921 further includes: receivinga user event input from a GUI displayed at the user device, the userevent input specifying one of the win or lose predictive resultsdisplayed to the user device as output; displaying sub-detail for one ofthe sales opportunities as updated output to the user device responsiveto the user event input; and in which the sub-detail includes one ormore features probabilistically related to the win or lose predictiveresults displayed.

According to another embodiment of method 1921, the one or more featuresprobabilistically related to the win or lose predictive resultsdisplayed includes one or more name=value pairs derived from the socialmedia data source specified and having affected the updated win or losepredictive results for each of the selected one or more salesopportunities.

According to another embodiment there is a non-transitory computerreadable storage medium having instructions stored thereon that, whenexecuted by a processor in a host organization, the instructions causethe host organization to perform operations including: receiving inputfrom a user device specifying a dataset of sales data for a customerorganization, in which the sales data specifies a plurality of salesopportunities; generating indices from rows and columns of the dataset,the indices representing probabilistic relationships between the rowsand the columns of the dataset; storing the indices in a queryabledatabase within the host organization; selecting one or more of theplurality of sales opportunities specified within the sales data;querying the indices for a win or lose predictive result for each of theselected one or more sales opportunities; and displaying the win or losepredictive result for each of the selected one or more salesopportunities to the user device as output.

FIG. 20A depicts a pipeline change report 2001 in accordance withdescribed embodiments. On the left, a pipeline change report showinghistorical sum of amounts 2002 across snapshot dates 2004 is depictedand on the right a pipeline change report showing historical recordcounts 2003 across snapshot dates 2004 is depicted, thus presenting auser with their open pipeline for the current month (e.g., the month ofJanuary 2013 here) arranged by sales stage inclusive of such stages onthe historical dates charted. For instance, such stages may include:perception analysis, proposal/price quote, and negotiation/review, etc.

The pipeline change report 2001 enables users to see their data in anaggregated fashion. Each stage may consist of multiple opportunities andeach is capable of being duplicated because each of the opportunitiesmay change according to the amounts or according to the stage, etc.Thus, if a user is looking at the last four weeks, then one opportunitymay change from $500 to $1500 and thus be duplicated.

The cloud computing architecture executes functionality which runsacross all the data for all tenants. Thus, for any cases, leads, andopportunities, the database maintains a historical trending data object(HTDO) into which all audit data is retained such that a full and richhistory can later be provided to the user at their request to show thestate of any event in the past, without corrupting the current state ofthe data stored on behalf of database tenants while allowing databaseupdates to be committed. Thus, while the underlying data must bemaintained in its correct state for the present moment, a user maynevertheless utilize the system to display the state of a particularopportunity as it historically stood, regardless of whether the datarequested is for the state of the opportunity last week, or as ittransitioned through the past quarter, and so forth.

All of the audit data from history objects for various categories ofdata is then aggregated into a historical trending data object. Thehistorical trending data object is then queried by the differenthistorical report types across multiple tenants to retrieve thenecessary audit trail data such that any event at any time in the pastcan be re-created for the sake of reporting, predictive analysis, andexploration. The historical audit data may additionally be subjected tothe analysis capabilities of the analysis engine (e.g., element 185 ofFIG. 1) by including the historical audit data within a historicaldataset for the sake of providing further predictive capabilities onthat data. For instance, while historical data is known for the variousopportunities, a future state can be predicted for those sameopportunities to aid the salespersons in focusing their effortsappropriately.

FIG. 20B depicts a waterfall chart using predictive data in accordancewith described embodiments. Opportunity count 2006 defines the verticalaxis and stages 2005 define the horizontal axis from “start” to “end”traversing stages 1 through 8. For instance, the waterfall chart maydepict a snapshot of all opportunities presently being worked broken outby stage. The opportunity counts change up and down by stage to reflectthe grouping of the various opportunities into the various definedstages. The waterfall chart may be used to look at two points bydefining opportunities between day one and day two or as is shown viathe example here. The waterfall chart may be used to group allopportunities into different stages in which every opportunity is mappedaccording to its present stage, thus allowing a user to look into thepast and understand what the timing was for these opportunities toactually come through to closure.

Historical data and the audit history saved to the historical trendingdata object are enabled through snapshots and field history. Using thehistorical trending data object the desired data can then be queried.The historical trending data object may be implemented as one table withindexes on the table from which any of the desired data can then beretrieved. The various specialized GUIs and use cases are populatedusing the opportunity data retrieved from the historical trending dataobject's table.

FIG. 20C depicts an interface with defaults after adding a firsthistorical field. Element 2011 depicts the addition of a historicalfield filter which includes various options including to filter byhistorical amounts (e.g., values in excess of $1 million), to filter bya field (e.g., account name equals Acme), to filter by logic (e.g.,filter 1 AND (filter 2 OR filter 3)), to cross filter (e.g., accountswith or without opportunities), to row limit (e.g., show only the top 5accounts by annual revenue), and finally, a “Help me choose” option.

The interface enables the user to filter historical data by comparinghistorical values versus current values stored within in themulti-tenant system.

FIG. 20D depicts in additional detail an interface with defaults for anadded custom filter. These specialized filtering implementations enableusers to identify how the data has changed on a day to day basis or weekto week basis or over a month to month basis, etc. The users cantherefore can see the data that is related to the user's opportunitiesnot just for the present time, but with this feature, the users canidentify opportunities based on a specified time such as absolute timeor relative time, so that they can see how the opportunity has changedover time. In this embodiment, time as a dimension is used to thenprovide a decision tree for the customers to pick either absolute dateor a range of dates. Customers can pick an absolute date, such as Jan.1, 2013 or a relative date such as the first day of the current month orthe first day of the last month, and so forth.

Menus may be populated exclusively with historical field filters and mayuse historical color coding as depicted by element 2025. At element 2026the selection has defaulted to rolling day in which “Any SelectedHistorical Date” may be selected. Alternatively, fixed days may beselected, but this option is collapsed by default in the depictedinterface. Element 2027 sets forth a variety of operators that may beselected by a user depending on the historical field type chosen, andelement 2028 provides a default amount value (e.g., $1,000,000) as aplaceholder attribute that is alterable by the user.

The custom filter interface depicted enables a sales manager orsalesperson to see how an opportunity has changed today versus the firstday of this month or last month, etc. Through the custom filterinterface, a user can take a step back in time, thinking back where theywere a week ago or a month ago and identify the opportunity by creatinga range of dates and displaying what opportunities were created duringthose dates.

For example, a salesperson wanting such information may have had tenopportunities and on Feb. 1, 2013, the salesperson's target buyerexpresses interest in a quote, causing the stage to change fromprospecting to quotation. Conversely, another target buyer says theywant to buy immediately, causing the state to change from quotation tosale/charge/close. The custom filter interface therefore provides adecision tree based on the various dates that are created, guiding auser through the input and selection process by only revealing theappropriate selections and filters for the dates initially selected. Theresult is that the functionality can give the salesperson a view of allthe opportunities that are closing in the month of January, or February,or within a given range, within a quarter or year, and so forth, in ahighly intuitive manner.

Querying by date necessitates the user to traverse the decision tree toidentify the user's desired date then enabling the user to additionallypick the number of snap shots, from which the finalized result set isdetermined, for instance, from Feb. 1, 2013 to Feb. 6, 2013.

Additionally enabled is the ability to filter historical data bycomparing historical values versus a constant in the multi-tenantsystem, referred to as a historical selector. Based on the opportunityor report type, the customer has the ability to filter on historicaldata using a custom historical filter. The interface provides theability for the customer to look at all of the filters on the left thatthey can use to restrict a value or a field, thus allowing customers tofilter on historical column data for any given value. Thus, a customermay look at all of the open opportunities for a given month or filterthe data set according to current column data rather than historical.Thus, for a given opportunity a user at the interface can fill out theamount, stage, close date, probability, forecast category, or other dataelements and then as the salesperson speaks with the target buyer, thestate is changed from prospecting to quoting, to negotiation based onthe progress that is made with the target buyer, and eventually to astate of won/closed or lost, etc. Filtering on elements such asprobability of close and forecast category will trigger predictivequeries to render the predictions upon which filtering and othercomparisons by the interface are made.

Take for example, a target buyer asking to decrease the amount of thedeal and a salesperson trying to increase the amount. All of the dataincluding changing amounts for the opportunity and state changes for theopportunity is stored in the historical trending data object whichprovides the audit trail.

As the current data is updated within the current tables past valuesbecome inaccessible to the customer. However, the historical trendingdata object provides a queryable audit trail through which suchhistorical values may be retrieved at the behest of the interface andits users. According to one embodiment, the historical data is processedwith granularity of one day, and thus, a salesperson can go back in timeand view how the data has changed overtime with within the data set withthe daily granular reporting. In other embodiments, all changes aretracked and time-stamped such that any change, no matter the frequency,can be revealed.

In addition to revealing how opportunities have changed over time onbehalf of salespersons, such metrics may be useful to other disciplinesalso, such as a service manager running a call center that receiveshundreds of cases from sales agents and needs to evaluate the best meansby which to close the calls. Likewise, campaign managers running amarketing campaign can evaluate the best means by which to close on thevarious leads and opportunities unveiled through the marketing effort aswell as peer back into history to see how events influenced the resultsof past opportunities.

FIG. 20E depicts another interface with defaults for an added customfilter. Here in the “amount” selector a “field” mode has been selectedrather than value as in the prior example depicted at FIG. 20D. Element2029 indicates that when in “field” mode, only current values willappear in the picker, thus permitting the user to select from amongthose values that actually exist within the date range restricted dataset, instead of entering a value. The interface is not limited to“amount” but rather, operates for any columns within a dataset and thenpermits filtering by value or by “field” mode in which the picker listsonly those values which exist in one or more fields for the specifiedcolumn. For instance, by selecting “stage” the picker may depict eachstage of a four stage process, assuming at least one opportunity existedin each of the exemplary four stages for the customer historical daterange specified. In such a way, the user is presented with a highlyintuitive interface by which to explore the historical data accessibleto them.

In other embodiments, filter elements are provided to the user to narrowor limit the search according to desired criteria, such as industry,geography, deal size, products in play, etc. Such functionality thusaids sales professionals with improving sales productivity andstreamlining business processes.

According to one embodiment, the historical trending data object isimplemented via a historical data schema in which historical data isstored in a table such as that depicted at Table 1 below:

TABLE 1 column name data type nullable notes organization_id char(15) nokey_prefix char(3) no key prefix of historical data itselfhistorical_entity_data_id char(15) no parent_id char(15) no FK to theparent record transaction_id char(15) no generated key used to uniquelyidentify transaction that changed the parent record. Main purpose is toreconcile multiple changes that may occur in one transaction (customfield versus standard field, for example may be written separately) andenable asynchronous fixer operations (if used). division number nocurrency_iso_code char(3) no deleted char(1) no row_version number nostandard audit fields valid_from date no with valid_to, defines timeperiod the data is valid. The time periods (valid_from, valid_to) foreach snapshot of the same parent does not overlap. Gaps are allowed.valid_to date no default to 3000/1/1 for current data val0 . . . val800varchar(765) yes flex fields for storing historic values

Indices utilized in the above Table 1 include: organization_id,key_prefix, historic_entity_data_id. PK includes: organization_id,key_prefix, system_modstamp (e.g., providing time stamping or a timestamped record, etc.). Unique, find, and snapshot for given date andparent record: organization_id, key_prefix, parent_id, valid_to,valid_from. Indices organization_id, key_prefix, valid_to facilitatedata clean up. Such a table is additionally counted against users'storage requirements according to certain embodiments. For example,usage may be capped at a pre-configured number of records per user ormay be alterable based on pricing plans for the user's organization.Alternatively, when available slots are running low, old slots may becleaned. Historical data management, row limits, and statistics may beoptionally utilized. For new history the system may assume an average 20byte per column and 60 effective columns (50 effective datacolumns+PK+audit fields) for the new history table, and thus, row sizeis 1300 bytes. For row estimates the system may assume that historicaltrending will have usage patterns similar to entity history. By charginghistorical trending storage usage to a customer's applicable resourcelimits, users and organizations will balance the depth of desiredhistorical availability against their resource constraints and pricing.

Sampling of production data revealed recent growth in row count forentity history is approximately 2.5 B (billion) rows/year. Sincehistorical trending will store a single row for any number of changedfields, an additional factor of 0.78 can be applied. By restricting thetotal quantity of custom historical trending data objects perorganization, an expected row count for historical trending may belimited to approximately 1.2 B rows per year in the worst case scenario,with pricing structures being used to influence the total collectivegrowth amongst all tenants of the database.

Historical data may be stored for a default number of years. Where twoyears is provided as an initial default, the size of the historicaltrending table is expected to stay around 2.4 B rows. Custom valuecolumns are to be handled by custom indexes similar to custom objects.To prevent unintentional abuse of the system, for example, by usingautomated scripts, each organization will have a history row limit foreach object. Such a limit may be between approximately 1 and 5 millionrows per object which is sufficient to cover storage of current data aswell as history data based on analyzed usage patterns of production datawith only very few organizations occasionally having so many objectsthat they may hit the configurable limit. Such limits may be handled ona case by cases basis while enabling reasonable limits for theoverwhelming user population. The customized table may additionally becustom indexed to help query performance for the various users into thehistorical trending data object.

FIG. 20F depicts an exemplary architecture in accordance with describedembodiments. In particular, customer organizations 2033A, 2033B, and2033C are depicted, each with a user's client device and display 2034A,2034B, and 2034C capable of interfacing with host organization 2010 vianetwork 2032, including sending input, queries, and requests andresponsively receiving responses including output for display. Withinhost organization 2010 is a request interface 2076 which may optionallybe implemented by web-server 2075. The host organization furtherincludes processor(s) 2081, memory 2082, a query interface 2080,analysis engine 2085, and a multi-tenant database system 2030. Withinthe multi-tenant database system 2030 are execution hardware, software,and logic 2020 that are shared across multiple tenants of themulti-tenant database system 2030, authenticator 2098, and databases2050 which may include, for example, a database for storing records,such as a relational database, a database for storing historical values,such as an object database capable of hosting the historical trendingdata object, and a predictive database capable of storing indices 2054generated by the analysis engine 2085 to facilitate the return ofpredictive results responsive to queries executed against such apredictive database.

According to one embodiment, the host organization 2010 operates asystem 2035 having at least a processor 2081 and a memory 2082 therein,in which the system 2035 includes a database 2050 to store records 2060,in which updates to the records 2060 are recorded into a historicaltrending data object (HTDO) 2061 to maintain historical values for therecords when the records 2060 are updated in the database 2050.According to such an embodiment, the system 2035 further includes arequest interface 2076 to receive input 2053 from a user device 2034A-Cspecifying data to be displayed at the user device 2034A-C and furtherin which the request interface 2076 is to receive historical filterinput 2056 from the user device 2034A-C. In such an embodiment, thesystem 2035 further includes a query interface 2080 to query 2057 therecords 2060 stored in the database 2050 for the data to be displayed2058 and further in which the query interface 2080 is to query 2057 thehistorical trending data object 2061 for the historical values 2062 ofthe data to be displayed 2058. The system 2035 further includes ananalysis engine 2085 to compare the data to be displayed with thehistorical values of the data to be displayed to determine one or morechanged values 2063 corresponding to the data to be displayed. Therequest interface 2076 of the system 2035 is then to further return, asdisplay output 2055 to the user device 2034A-C, at least the data to bedisplayed 2058 and a changed value indication based on the one or morechanged values 2063 determined via the comparing.

The request interface 2076 may additionally receive selection input 2065via a change value indication GUI at the user device 2034A-C, in whichthe selection input 2065 requests additional sub-detail for the one ormore changed values, responsive to which the request interface 2076and/or web-server 2075 may provide additional drill-down sub-detail. Forinstance, if a user of a touchscreen touches or gestures to one of thechanged values indicated then the user interface may communicate suchinput to the request interface 2076 causing the host organization 2010to provide updated display output 2059 with additional detail for thespecified changed value (e.g., present and past state, difference,direction of change, predictive win/loss result change for a salesopportunity, etc.).

According to another embodiment of the system 2035, the databases 2050are to execute as on-demand cloud based services at the hostorganization 2010 for one or more subscribers; and in which the systemfurther includes an authenticator 2098 to verify the user as a knownsubscriber and to further verify authentication credentials presented bythe known subscriber.

According to another embodiment of the system 2035, a web-server 2075 isto implement the request interface 2076 and is to interact with a changevalue indication GUI caused to be displayed at the user device 2034A-Cby the request interface 2076 and/or web-server 2075. In such anembodiment, the web-server 2075 is to receive as input, a plurality ofaccess requests from one or more client devices from among a pluralityof customer organizations communicably interfaced with the hostorganization via a network 2032.

The system 2035 may further include a multi-tenant database system withpredictive database functionality to implement the predictive database;and further in which each customer organization is an entity selectedfrom the group consisting of: a separate and distinct remoteorganization, an organizational group within the host organization, abusiness partner of the host organization, or a customer organizationthat subscribes to cloud computing services provided by the hostorganization.

FIG. 20G is a flow diagram illustrating a method in accordance withdisclosed embodiments. Method 2031 may be performed by processing logicthat may include hardware (e.g., circuitry, dedicated logic,programmable logic, microcode, etc.), software (e.g., instructions runon a processing device to perform various operations such transmitting,sending, receiving, executing, generating, calculating, storing,exposing, querying, processing, etc., in pursuance of the systems,apparatuses, and methods for implementing change value indication andhistorical value comparison at a user interface, as described herein.For example, host organization 110 of FIG. 1, machine 400 of FIG. 4, orsystem 2035 of FIG. 20F may implement the described methodologies. Someof the blocks and/or operations listed below are optional in accordancewith certain embodiments. The numbering of the blocks presented is forthe sake of clarity and is not intended to prescribe an order ofoperations in which the various blocks must occur.

At block 2091, processing logic stores records in a database, in whichupdates to the records are recorded into a historical trending dataobject to maintain historical values for the records when the recordsare updated in the database.

At block 2092, processing logic receives input from a user devicespecifying data to be displayed at the user device.

At block 2093, processing logic receives historical filter input fromthe user device.

At block 2094, processing logic queries the records stored in thedatabase for the data to be displayed.

At block 2095, processing logic queries the historical trending dataobject for the historical values of the data to be displayed.

At block 2096, processing logic compares the data to be displayed withthe historical values of the data to be displayed to determine one ormore changed values corresponding to the data to be displayed.

At block 2097, processing logic displays a change value indication GUIto the user device displaying at least the data to be displayed and achanged value indication based on the one or more changed valuesdetermined via the comparing.

The User Interface (UI) or Graphical User Interface (GUI) and the changevalue indication GUI in particular consumes data stored in the database,consumes historical data stored within the historical trending dataobject, and consumes predictive results returned from the predictiveinterface to display results to the user in a highly intuitive fashion.

The problem with conventional database interfaces is that they view datastored within the database in its present state, which is, of course,the objective of a database that stores records. Nevertheless, it issometimes beneficial to have a view of the data as it was on somehistorical date in the past, or have a view of how the data has changedbetween two historical dates or between a past state on a particularhistorical date and a current state as the data exists today.

Recovering data in the database to a prior state is not a workablesolution as this will overwrite the data in its present state witherroneous past state data. Accordingly, the change value indication GUIprovides an intuitive means by which a user can explore their data, evenas it existed in a past state. Such capabilities allow a user to stepback in time and view the data, such as the current state of a salesopportunity or other such records, as it existed on a specified day,without corrupting the up-to-date date in its present state as storedwithin a database.

Moreover, the described methodologies negate the need for a user todefine complex data schemas or write custom code to track historicaldata. For instance, there is no need to engage IT support to expose thenecessary data or employ programmers to write customized software totrack such information. Instead, a host organization operating as acloud based service provides the necessary functionality to the user andexposes it through an intuitive UI, such as the change value indicationGUI described.

Further still, users are not required to construct complicated SQLqueries, but rather, may explore and view historical data records andvalues through the GUI interface. When the change value indication GUIis coupled with predictive queries, the GUI constructs the necessaryPreQL queries on behalf of the user and exposes predictive results tothe GUI, thus further expanding data exploration capabilities for theuser.

According to another embodiment of method 2031, the records stored inthe database maintain a present state for the data; and in which thehistorical values for the records recorded into the historical trendingdata object maintain one or more past states for the data withoutcorrupting the present state of the data.

According to another embodiment of method 2031, storing the records inthe database includes storing one or more tables in the database, eachof the one or more tables having a plurality of columns establishingcharacteristics of entities listed in the table and a plurality ofentity rows recorded in the table as records; and in which updates tothe records include any one of: (i) modifying any field at an intersectof the plurality of columns and the plurality of entity rows, and (ii)adding or deleting a record in the database.

According to another embodiment of method 2031, updates to the recordsincludes: receiving an update to a record stored in the database;recording present state data for the record stored in the database intothe historical trending data object as past state data; modifying thepresent state data of the record in the database according to the updatereceived; committing the update to the database; and committing the paststate data to the historical trending data object.

According to another embodiment of method 2031, the historical valuesfor the records maintained in the historical trending data object aretime stamped; and in which multiple updates to a single record stored inthe database are distinctly maintained within the historical trendingdata object and differentiated based at least on the time stamp. Incertain embodiments, every update to a record within the database isstored as a new row within the historical trending data object.

The computing hardware of the host organization thus stores every changethat occurs within the database records on behalf of users as raw datawithin the historical trending data object and when users engage the GUIinterface, appropriate queries are constructed on behalf of the user toquery and retrieve the necessary information to display changed values,to display differences in values, to display changes over time for agiven field, and so forth. For example, a user may specify two points intime, such as today and last month, from which the necessary functionsare built by the GUI's or web-server's functionality to query anddisplay the results to the user via the change value indication GUI,including computed differences or modified values, along withhighlighting to emphasize determined changes in values.

For example, the change value indication GUI may display undesirablechanges as red text, with red directional arrows, or with redhighlighting. Such undesirable changes may be a reduction in a salesopportunity amount, a reduction in probability of a predictive win|loseresult (e.g., a predictive result for an IS_WON field having a nullvalue), an increase in the predicted sales opportunity close date, andso forth. Desirable changes may be displayed using green text, arrows,or highlighting, and changes that are neutral may simply use gray orblack text, arrows, or highlighting. Different colors than thosedescribed may be substituted.

According to another embodiment of method 2031, the changed valueindication includes at least one of: colored or highlighted textdisplayed at the change value indication GUI for the one or more changedvalues; directional arrows displayed at the change value indication GUIfor the one or more changed values, the directional arrows indicating anexistence of change or a direction of change; a computed differencebetween the data to be displayed and the one or more changed values; anda present state and a past state displayed concurrently for the data tobe displayed based on the one or more changed values.

According to another embodiment of method 2031, displaying a changevalue indication GUI to the user device includes displaying both thedata to be displayed and the one or more changed values determined viathe comparing in addition to the changed value indication.

According to another embodiment of method 2031, the data to be displayedincludes a plurality of sales opportunities stored as the records in thedatabase; and in which displaying the change value indication GUI to theuser device includes displaying the plurality of sales opportunities tothe user device with the changed value indication depicting changes toone or more of the plurality of sales opportunities in a current stateversus a past state.

According to another embodiment, method 2031 further includes:determining a first win or lose predictive result for each of the salesopportunities in the current state; determining a second win or losepredictive result for each of the sales opportunities in the past state;and depicting any change between the first and second win or losepredictive results via the changed value indication GUI.

According to another embodiment of method 2031, determining the firstand second win or lose predictive results includes constructing apredictive query specifying a PREDICT command term and issuing thepredictive query against a predictive database via a Predictive QueryLanguage (PreQL) interface.

According to another embodiment of method 2031, the change valueindication GUI depicts a graph or chart of the one or more changedvalues over time based on the historical values of the data to bedisplayed.

The change value indication GUI permits users to customize reports via avariety of filters including both normal filters that filter results ofpresent state data stored within the database as well as historicalfilters. Utilizing the GUI, users may add a new filter for a report andspecify, for example, a historical field filter along with logicalcomparators (e.g., equal to, less than, greater than, is true, is false,is null, etc). The user may additionally specify historical dates foruse in filtering. For instance, results may be requested as they existedon a given historical date, or two dates may be specified which willthen yield changed values between the two dates, be they both historicaldates or a historical date compared to a present date (e.g., today). Adate range is sometimes appropriate, for instance, to show how a salesamount has changed over time on a month by month, week by week, or dayby day basis, and so forth.

According to another embodiment of method 2031, the historical filterinput includes at least one of: a historical date; a historical daterange; a historical close date for a closed sales opportunity; a valueor string recorded in the historical trending data object; a field orrecord present in the historical trending data object; a logical operandor comparator for a value or string recorded in the historical trendingdata object; and a predictive result threshold or range for null valuespresent in the historical trending data object.

According to another embodiment of method 2031, displaying the changevalue indication GUI to the user device further includes at least oneof: displaying all changed values for the data to be displayeddeterminable from the historical trending data object; displaying allchanged values within a date range specified via the historical filterinput; and displaying a graph of the one or more changed values with adaily, weekly, monthly, or quarterly change interval as specified viathe change value indication GUI.

According to another embodiment of method 2031, the historical trendingdata object is to maintain the historical values is active by defaultand exposed to users via the change value indication GUI as part of acloud computing service.

According to another embodiment of method 2031, the historical trendingdata object is limited to a historical capacity established based onsubscription fees paid by the users for access to the cloud computingservice, the historical capacity increasing in proportion to thesubscription fees paid by the users, with zero subscription fee usershaving access to a minimum default historical capacity.

Such a model encourages users to maintain their existing data within thecloud at the host organization because users are able to benefit fromenhanced capabilities which are not provided by conventional solutions.Even where users do not pay additional fees, they are still exposed tothe capability in a limited fashion and can decide later whether or notthey wish to expand the scope of their historical data exploration andretention capabilities.

According to another embodiment, method 2031 further includes:displaying additional sub-detail for the one or more changed valuesresponsive to selection input received at the change value indicationGUI; in which the selection input includes one of a mouse over event, acursor over event, a click event, a touchscreen selection event, or atouchscreen position event corresponding the change value indicationdisplayed; and in which displaying sub-detail includes displaying thesub-detail within a graphical overlay positioned on top of and at leastpartially covering the change value indication displayed initially.

For example, having returned the display output to the user's display,the user may further explore the results by clicking, gesturing,pressing, or hovering on an item, which then triggers the change valueindication GUI to render additional results contextually relevant to theuser's actions without requiring the user to construct alternative oradditional filtering.

According to another embodiment of method 2031, the one or more changedvalues correspond to at least one of: (i) a change in a win or losepredictive result indicating whether a sales opportunity isprobabilistically predicted to result in a win or a loss; (ii) a changein a confidence indicator for the win or lose predictive result; (iii)change in a predicted sales amount; (iv) a change in a predicted salesopportunity close date; and in which sub-detail corresponding to any oneof (i) through (iv) is further displayed to the user device via thechange value indication GUI responsive to selection input received atthe change value indication GUI.

The ability to compare historical results enables a pipeline comparison(e.g., refer to the pipeline change report 2001 at FIG. 20A). Such areport enables users to explore how products, sales opportunities, orother natural business flows change over time, for instance, how thebusiness pipeline looks today versus yesterday, or today versus lastquarter, or how the business pipeline looked at the end of last quarterversus the same quarter of the prior year, and so forth. Such a reportmay depict sales opportunities charted against sales stages, sales byproduct, forecast, category, predictive results for yet to be observedvalues, volumes, revenues, and any other business metric for which datais recorded in the database or capable of prediction via the predictivedatabase.

According to another embodiment of method 2031, displaying the changevalue indication GUI to the user includes: displaying a first pipelinechart to the user device defined by quantity of sales opportunities on afirst axis against a plurality of available sales stages for the salesopportunities on a second axis on a historical date specified via thechange value indication GUI based on the historical values maintainedwithin the historical trending data object; and displaying a secondpipeline chart concurrently with the first pipeline chart, the secondpipeline chart depicting the quantity of sales opportunities against theplurality of available sales stages on a second historical date or acurrent date as specified via the change value indication GUI.

According to another embodiment of method 2031, the change valueindication GUI further displays a recommended forecasting adjustmentbased on predictive analysis of the historical values maintained withinthe historical trending data object by an analysis engine.

According to another embodiment of method 2031, the change valueindication depicts the one or more changed values using red text orhighlighting, green text or highlighting, and gray text or highlighting;in which the red text or highlighting represents a negative change of apresent state versus a past state determined via the comparing; in whichthe green text or highlighting represents a positive change of a presentstate versus a past state determined via the comparing; and in which thegray text or highlighting represents a neutral change of a present stateversus a past state determined via the comparing.

According to another embodiment there is a non-transitory computerreadable storage medium having instructions stored thereon that, whenexecuted by a processor in a host organization, the instructions causethe host organization to perform operations including: storing recordsin a database, in which updates to the records are recorded into ahistorical trending data object to maintain historical values for therecords when the records are updated in the database; receiving inputfrom a user device specifying data to be displayed at the user device;receiving historical filter input from the user device; querying therecords stored in the database for the data to be displayed; queryingthe historical trending data object for the historical values of thedata to be displayed; comparing the data to be displayed with thehistorical values of the data to be displayed to determine one or morechanged values corresponding to the data to be displayed; and displayinga change value indication GUI to the user device displaying at least thedata to be displayed and a changed value indication based on the one ormore changed values determined via the comparing.

FIG. 21A provides a chart depicting prediction completeness versusaccuracy. On the vertical axis at element 2105 accuracy/confidence isshown ranging from “1.0” representing essentially perfect accuracy orthe highest possible confidence in a prediction down to “0.4” on thisparticular scale, representing somewhat poor accuracy or low confidence.On the horizontal axis, element 2106 depicts filler percentage rangingfrom “0.0” meaning there is no predictive fill to “1.0,” meaning allavailable elements are filled using predictive results where necessary.Thus, at 0.0, there are no predicted results and as such, accuracy isperfect because only known (e.g., actually observed) data is present.Conversely, at 1.0 fill percentage, predictive results become lessreliable, such that any null-values present in a data set are filledusing predictive values, but with accuracy/confidence reaching a lowbetween 0.4 and 0.5.

Any number of different intersections can be drawn, however, element2107 depicts the intersection between 0.8 accuracy/confidence on thevertical axis and above 50% fill percentage on the horizontal axis whichtranslates to sales predictions being 80% accurate/confident for greaterthan 50% of the opportunities analyzed by the predictive analysisengine's core.

Different datasets may change these precise values, however, the chartdepicts what is commonly found within rich datasets pertaining to suchsales data. Specifically that a majority of opportunities can bepredicted with a relatively high degree of accuracy/confidence, which inturn permits the salespersons to focus their efforts on thoseopportunities which are most likely to yield a positive result,according to the predictive analysis performed.

FIG. 21B provides a chart depicting an opportunity confidence breakdown.Element 2011 on the vertical axis depicts the number of opportunitiesranging from 0 to 9000 on this particular chart and element 2012 on thehorizontal axis represents the probability of sale, ranging from a 0.0confidence to a 1.0 confidence. Notably, the columns toward the left andalso the columns toward the right are highly revealing. A probability of“0.0” does not correlate to complete lack of confidence, but rather,correlates to a very high degree of confidence that the sales are highlyunlikely to result in a sale as depicted by element 2013 highlightingthose sales opportunities ranting from 0.0 to 0.2. On the opposite endof the spectrum, element 2014 highlights those sales opportunitiesranging from 0.8 to 1.0 as being highly likely to close in a sale.

FIG. 21C provides a chart depicting an opportunity win prediction. Insignal detection theory, a Receiver Operating Characteristic (ROC), orsimply a ROC curve, is a graphical plot which illustrates theperformance of a binary classifier system as its discriminationthreshold is varied. It is created by plotting the fraction of truepositives out of the positives (TPR=true positive rate) vs. the fractionof false positives out of the negatives (FPR=false positive rate) atvarious threshold settings. TPR is also known as sensitivity (alsoreferred to as recall in some fields), and FPR is one minus thespecificity or true negative rate. In general, if both of theprobability distributions for detection and false alarm are known, theROC curve can be generated by plotting the Cumulative DistributionFunction (area under the probability distribution from −infinity to+infinity) of the detection probability in the y-axis versus theCumulative Distribution Function of the false alarm probability inx-axis.

The ROC 10 k curve depicted here maps the True Positive Rate on thevertical axis marked by element 2021 ranging from a confidence of 0.0 to1.0 and further maps the False Positive Rate on the horizontal axismarked by element 2022 ranging from a confidence of 0.0 to 1.0 resultingin a ROC curve having an area of 0.93.

FIG. 22A provides a chart depicting predictive relationships foropportunity scoring. Predictive currency is conditioned on the “ISWON=True|False” field. Element 2206 on the vertical axis depicts “ISWON” as True or False by source and element 2207 on the horizontal axisdepicts a variety of sales lead sources including from left to right,website, Salesforce AE, Other, EBR Generated, Sales Generated, PartnerReferral, AE/Sales, Internet Search—Paid, Inbound Call, and lastly AEGenerated—Create Account on the right.

The interface depicted here is generated on behalf of a user using ahistorical data set subjected to predictive analysis and may aid asalesperson or sales team in determining where to apply limitedresources. The chart is subject to interpretation, but certain facts arerevealed by the analysis such as element 2213 which indicates that EBRGenerated leads are highly likely to win a sale, element 2212 depictsthat AE/Sales are less likely to win a sale, and at element 2211 it canbe seen that Inbound Calls result in about a 50/50 chance to win a sale.Such data presented at the interface showing predictive relationshipsfor opportunity scoring may thus be helpful to a sales team indetermining where to focus resources.

FIG. 22B provides another chart depicting predictive relationships foropportunity scoring. Here the opportunity is conditioned on the “ISWON=True|False” field. Element 2221 on the vertical axis depicts “ISWON” as True or False by type and element 2222 on the horizontal axisdepicts a variety of sales lead types including from left to right,Add-On Business, New Business, Public, Renewal, and Contract on theright.

Element 2223 depicts that Add-On business is more likely to win a saleand element 2224 indicates that New Business is less likely to win asale. As before, the interface depicted here is generated on behalf of auser using a historical data set subjected to predictive analysis andmay aid a salesperson or sales team in determining where to applylimited resources.

Additional functionality enables specialized UI interfaces to render alikelihood to renew an existing opportunity by providing a score orprobability of retention for an existing opportunity by providing aretention score. Such functionality is helpful to sales professionals assuch metrics can influence where a salesperson's time and resources arebest spent so as to maximize revenue.

Opportunity scoring may utilize the RELATED command term to issue alatent structure query request to indices generated by the analysisengine's predictive analysis of a dataset. For instance, the RELATEDcommand term may be utilized by a specialized UI to identify whichfields are predictively related to another field, such as which fieldsare related to an “IS WON” field with true or false values. Other lessintuitive fields may additionally be probabilistically related. Forinstance, a lead source field may be determined to be related to certaincolumns of the dataset whereas other columns such as the fiscal quartermay prove less related to a win/loss outcome.

FIG. 22C provides another chart depicting predictive relationships foropportunity scoring. Here the predicted currency is conditioned on the“IS WON=True|False” field. Element 2231 on the vertical axis depicts “ISWON” as True or False by currency and element 2232 on the horizontalaxis depicts a variety of sales leads by currency including from left toright, United States Dollars (USD), Australian Dollars (AUD), JapaneseYen (JPY), Great British Pounds (GBP), Canadian Dollars (CAD), andlastly Euros (EUR) on the right.

Interpreting the data, it can be said at element 2236 that opportunitiesare more likely to result in a sales win in Japan and at element 2237opportunities are less likely to win in European countries using theEuro. While significantly more data exists for USD based salesopportunities, there is less of a clear relationship to win/loss bycurrency, although slightly more sales analyzed resulted are predictedto result in a win versus a loss. As before, the interface depicted hereis generated on behalf of a user using a historical data set subjectedto predictive analysis and may aid a salesperson or sales team indetermining where to apply limited resources.

High level use cases for such historical based data in a dataset to beanalyzed and subjected to predictive analysis are not limited to theexplicitly depicted examples. For instance, other use cases may include:determining a propensity to buy and scoring/ranking leads for salesrepresentatives and marketing users. For instance, sales users often getleads from multiple sources (marketing, external, sales prospectingetc.) and often times, in any given quarter, they have more leads tofollow up with than time available to them. Sales representatives oftenneed guidance with key questions such as: which leads have the highestpropensity to buy, what is the likelihood of a sale, what is thepotential revenue impact if this lead is converted to an opportunity,what is the estimated sale cycle based on historical observations ifthis lead is converted to an opportunity, what is the score/rank foreach lead in the pipeline so that high potential sales leads in asalesperson's territory may be discovered and prioritized, and so forth.

Sales representatives may seek to determine the top ten products eachaccount will likely buy based on the predictive analysis and the dealsizes if they successfully close, the length of the deal cycle based onthe historical trends of similar accounts, and so forth. When salesrepresentatives act on these recommendations, they can broaden theirpipeline and increase their chance to meet or exceed quota, thusimproving sales productivity, business processes, prospecting, and leadqualification. The historical data provided and subjected to predictiveanalysis may yield better predictive results which may be conveyed to auser through data exploration using the various filters or throughspecialized UI charts and interfaces provided, each of which handle thenecessary historical and predictive queries to the predictive databaseindices on behalf of the user.

Additional use cases for such historical based data may further include:likelihood to close/win and opportunity scoring. For instance, salesrepresentatives and sales managers may benefit from such data as theyoften have too many deals in their current pipeline and must jugglewhere to apply their time and attention in any month/quarter. As thesesales professionals approach the end of the sales period, the pressureto meet their quota is of significant importance. Opportunity scoringcan assist with ranking the opportunities in the pipeline based on theprobability of such deals to close, thus improving the overalleffectiveness of these sales professionals.

Additional data may be subjected to the predictive analysis along withhistorical sales data. Additional data sources may include such data as:comments, sales activities logged, standard field numbers for activities(e.g., events, log a call, tasks etc.), C-level customer contacts,decision maker contacts, close dates, standard field numbers for timesthe close date has pushed, opportunity competitors, standard fieldopportunities, competitive assessments, executive sponsorship, standardfield sales team versus custom field sales team as well as the membersof the respective teams, chatter feed and social network data for theindividuals involved, executive sponsor involved in a deal, DSRs (DealSupport Requests), and other custom fields.

Historical based data can be useful to the analysis engine's predictivecapabilities for generating metrics such as Next Likelihood Purchase(NLP) and opportunity whitespace for sales representatives and salesmanagers. For instance, a sales representative or sales managerresponsible for achieving quarterly sales targets will undoubtedly beinterested in: which types of customers are buying which products; whichprospects most resemble existing customers; are the right products beingoffered to the right customer at the right price; what more can we sellto my customer to increase the deal size, and so forth. Analyzinghistorical data for opportunities with similar customers known to havepurchased may uncover selling trends, and using such metrics yieldsvaluable insights to make predictions about what customers may buy next,thus improving sales productivity and business processes.

Another capability provided to end users is to provide customerreferences on behalf of sales professionals and other interestedparties. When sales professionals require customer references forpotential new business leads they often spend significant time searchingthrough and piecing together such information from CRM sources such ascustom applications, intranet sites, or reference data captured in theirdatabases. However, the analysis engine's core and associated use caseGUIs can provide key information to these sales professionals. Forinstance, the application can provide data that is grouped according toindustry, geography, size, similar product footprint, and so forth, aswell as provide in one place what reference assets are available forthose customer references, such as customer success stories, videos,best practices, which reference customers are available to chat with apotential buyer, customer reference information grouped according to thecontact person's role, such as CIO, VP of sales, etc., which referencecustomers have been over utilized and thus may not be good candidatereferences at this time, who are the sales representatives or accountrepresentatives for those reference customers at the present time or atany time in the past, who is available internally to an organization toreach out or make contact with the reference customer, and so forth.This type of information is normally present in database systems but isnot organized in a convenient manner resulting in an extremely laborintensive process to retrieve the necessary referral. However, theanalysis engine's core may identify such relationships and hiddenstructure in the data which may then be retrieved and displayed byspecialized GUI interfaces for end-users, for example, by calling theGROUP command term via the GUI's functionality. Additionally, thefunctionality can identify the most ideal or the best possible referencecustomer among many based on predictive analysis and incorporate thedetails of using a proposed reference customer into a scored probabilityto win/close opportunity chart. Such data is wholly unavailable fromconventional systems.

According to other embodiments, functionality is provided to predictforecast adjustments on behalf of sales professionals. For instance,businesses commonly have a system of sales forecasting as part of theircritical management strategy. Yet, such forecasts are by their verynature inexact. The difficulty is knowing in which direction suchforecasts are wrong and then turning that understanding into an improvedpicture of how the business is doing. The analysis engine's predictiveanalysis can improve such forecasting using a customer organization'sexisting data including existing forecasting data. For instance,analyzing past forecasting data in conjunction with historical salesdata may aid the business with trending and with improving existingforecasts into the future which have yet to be realized. Sales managersare often asked to provide their judgment or adjustment on forecastingdata for their respective sales representatives. Such activity requiressuch sales managers to aggregate their respective sales representatives'individual forecasts which is a very labor intensive process and tendsto introduce error. Sales managers are intimately familiar with theirrepresentatives' deals and they spend time reviewing them on a periodicbasis as part of a pipeline assessment. Improved forecasting results canaid such managers with improving the quality and accuracy of theirjudgments and assessments of current forecasting data as well as helpwith automating the aggregating function which is often carried outmanually or using inefficient tools, such as spreadsheets, etc.

In such an embodiment, the analysis engine mines past forecast trends bythe sales representatives for relationships and causations such asforecast versus quota versus actuals for a past time span, such as thepast eight quarters or other appropriate time period for the business.Using the analysis engine's predictive functionality or specialized UIinterfaces, a recommended judgment and/or adjustment is provided thatcan be applied to a current forecast. By leveraging the analyticalassessment at various levels of the forecast hierarchy, organizationscan reduce the variance between individual sales representative'sstipulated quotas, forecasts, and actuals, over a period of time,thereby narrowing deltas between forecast and realized sales viaimproved forecast accuracy.

While the subject matter disclosed herein has been described by way ofexample and in terms of the specific embodiments, it is to be understoodthat the claimed embodiments are not limited to the explicitlyenumerated embodiments disclosed. To the contrary, the disclosure isintended to cover various modifications and similar arrangements as areapparent to those skilled in the art. Therefore, the scope of theappended claims are to be accorded the broadest interpretation so as toencompass all such modifications and similar arrangements. It is to beunderstood that the above description is intended to be illustrative,and not restrictive. Many other embodiments will be apparent to those ofskill in the art upon reading and understanding the above description.The scope of the disclosed subject matter is therefore to be determinedin reference to the appended claims, along with the full scope ofequivalents to which such claims are entitled.

What is claimed is:
 1. A method in a host organization, the methodcomprising: receiving a tabular dataset from a user as input, thetabular dataset having data values organized as columns and rows;identifying a plurality of null values within the tabular dataset, thenull values being dispersed across multiple rows and multiple columns ofthe tabular dataset; generating indices from the tabular dataset ofcolumns and rows, the indices representing probabilistic relationshipsbetween the rows and the columns of the tabular dataset; displaying thetabular dataset as output to the user, the displayed output includingthe data values depicted as known values and the null values depicted asunknown values; receiving input from the user to populate at least aportion of the unknown values within the displayed tabular dataset withpredicted values; querying the indices for the predicted values; anddisplaying the predicted values as updated output to the user.
 2. Themethod of claim 1: wherein generating indices from the tabular datasetof columns and rows further comprises storing the indices within adatabase of the host organization; and wherein querying the indices forthe predicted values comprises querying the database for the predictedvalues.
 3. The method of claim 1: wherein receiving input from the userto populate at least a portion of the unknown values within thedisplayed tabular dataset with predicted values comprises receivinginput from the user to populate all unknown values within the displayedtabular dataset with predicted values; wherein querying the indices forthe predicted values comprises querying the indices for a predictedvalue for every null value identified within the tabular dataset; andwherein displaying the predicted values as updated output to the usercomprises replacing all unknown values by displaying correspondingpredicted values.
 4. The method of claim 1: wherein the plurality ofnull values within the tabular dataset are not restricted to any row orcolumn of the tabular dataset; and wherein displaying the predictedvalues as updated output to the user replaces the unknown valuesdisplayed with the tabular dataset without restriction to any row orcolumn and without changing the indices upon which the predicted valuesare based.
 5. The method of claim 1: wherein querying the indices forthe predicted values comprises querying the indices for each and everyone of the identified plurality of null values within the tabulardataset; wherein the method further comprises receiving the predictedvalues for each and every one of the identified plurality of null valueswithin the tabular dataset responsive to the querying; and whereindisplaying the predicted values as updated output to the user comprisesdisplaying the received predicted values.
 6. The method of claim 1,wherein querying the indices for the predicted values comprises:generating a Predictive Query Language (PreQL) query specifying aPREDICT command term for each and every one of the identified pluralityof null values within the tabular dataset; issuing each of the generatedPreQL queries to a Predictive Query Language Application ProgrammingInterface (PreQL API); and receiving a predicted result for each andevery one of the identified plurality of null values within the tabulardataset responsive to the issued PreQL queries.
 7. The method of claim1, wherein displaying the tabular dataset further comprises: displayingthe known values using black text within cells of a spreadsheet;displaying the unknown values as blank cells within the spreadsheet; anddisplaying the predicted values using colored or grayscale text withinthe cells of the spreadsheet.
 8. The method of claim 1: whereindisplaying the predicted values as updated output to the user comprisesdisplaying the updated output within a spreadsheet or table at aGraphical User Interface (GUI); wherein the known values are displayedas populated cells within the spreadsheet or table at the GUI in a firsttype of text; wherein predicted values are displayed as populated cellswithin the spreadsheet or table at the GUI in a second type of textdiscernable from the first type of text corresponding to the knownvalues; and wherein any remaining unknown values are displayed as emptycells within the spreadsheet or table at the GUI.
 9. The method of claim1: wherein displaying the tabular dataset as output to the user anddisplaying the predicted values as updated output to the user comprisesdisplaying the tabular dataset and the predicted values within aspreadsheet or table at a Graphical User Interface (GUI); wherein theGUI further comprises a slider interface controllable by the user tospecify an acceptable degree of uncertainty for the spreadsheet ortable; and wherein receiving input from the user to populate at least aportion of the unknown values within the displayed tabular dataset withpredicted values comprises receiving the acceptable degree ofuncertainty as input from the user via the slider interface.
 10. Themethod of claim 9, further comprising: displaying a minimum fillpercentage for the GUI, wherein the minimum fill percentage correspondsto a percentage of known values within the tabular dataset from a sum ofall null values and all known values for the tabular dataset.
 11. Themethod of claim 10, wherein the slider interface controllable by theuser to specify the acceptable degree of uncertainty for the spreadsheetor table is restricted to a range encompassing the minimum fillpercentage and a maximum degree of uncertainty necessary to completelypopulate the displayed tabular dataset.
 12. The method of claim 11,further comprising: populating the spreadsheet or table of the GUI to a100% fill percentage responsive to input by the user specifying amaximum acceptable degree of uncertainty via the slider interface; andpopulating all null values by degrading a required confidence for eachof the predicted values until a predicted value is available for everyone of the plurality of null values within the tabular dataset.
 13. Themethod of claim 1, further comprising: displaying a user controllableminimum confidence threshold at a Graphical User Interface (GUI)displaying the tabular dataset as output to the user within aspreadsheet or table; receiving a user specified minimum confidencethreshold as input via the user controllable minimum confidencethreshold; and wherein displaying the predicted values as updated outputto the user comprises displaying only the predicted values at the GUIhaving a corresponding confidence indicator equal to or greater than theuser specified minimum confidence threshold.
 14. The method of claim 1,further comprising: displaying a user controllable minimum confidencethreshold at a Graphical User Interface (GUI) displaying the tabulardataset as output to the user within a spreadsheet or table; anddisplaying a maximum fill percentage for the GUI, wherein the maximumfill percentage corresponds to a sum of all known values and all nullvalues returning a predicted value with a confidence indicator in excessof the user controllable minimum confidence threshold as a percentage ofa sum of all null values and all known values.
 15. The method of claim1, further comprising: receiving a confidence indicator for every one ofthe plurality of null values within the tabular dataset responsive toquerying the indices for the predicted values; and wherein displayingthe predicted values as updated output to the user comprises displayingselected ones of the predicted values that correspond to a confidencequality in excess of a default minimum confidence threshold or a userspecified minimum confidence threshold when present.
 16. The method ofclaim 1, further comprising: receiving a distribution for every one ofthe plurality of null values within the tabular dataset responsive toquerying the indices for the predicted values; calculating a credibleinterval for each distribution received; and wherein displaying thepredicted values as updated output to the user comprises displayingselected ones of the predicted values that correspond to a calculatedcredible interval in excess of a minimum threshold.
 17. The method ofclaim 1, wherein displaying the tabular dataset further comprises:displaying the known values as a first text type within cells of aspreadsheet; querying the indices for a predicted value corresponding toevery one of the unknown values; and wherein displaying the predictedvalues as updated output to the user comprises displaying the predictedvalues as a second text type within the cells of the spreadsheet,wherein the second text type has a displayed opacity in proportion to aconfidence indicator for the predicted value displayed.
 18. The methodof claim 1: wherein displaying the tabular dataset as output to the usercomprises displaying the known values as black text within cells of aspreadsheet; and wherein displaying the predicted values as updatedoutput to the user comprises displaying the predicted values asgrayscale text with the predicted values having a higher confidenceindicator being displayed at darker grayscales than the predicted valueshaving a lower confidence indicator.
 19. The method of claim 1, furthercomprising: displaying a prediction difficulty score for every column ofthe tabular dataset displayed as output to the user on a per-columnbasis, wherein the prediction difficulty score is calculated for eachcolumn of the tabular dataset by: (i) identifying all unknown valueswithin the column; (ii) querying the indices for a predicted valuecorresponding to each of the unknown values identified within thecolumn; (iii) receiving a confidence indicator for each of the unknownvalues identified within the column; and (iv) calculating the predictiondifficulty score for the column based on the confidence indicatorsreceived for the unknown values identified within the column.
 20. Themethod of claim 19, wherein the method further comprises: displaying amaximum fill percentage for every column of the tabular datasetdisplayed as output to the user on a per-column basis, wherein themaximum fill percentage is based on a quantity of the unknown valuesidentified within the respective column having confidence indicatorsexceeding a minimum confidence quality threshold.
 21. Non-transitorycomputer readable storage medium having instructions stored thereonthat, when executed by a processor in a host organization, theinstructions cause the host organization to perform operationscomprising: receiving a tabular dataset from a user as input, thetabular dataset having data values organized as columns and rows;identifying a plurality of null values within the tabular dataset, thenull values being dispersed across multiple rows and multiple columns ofthe tabular dataset; generating indices from the tabular dataset ofcolumns and rows, the indices representing probabilistic relationshipsbetween the rows and the columns of the tabular dataset; displaying thetabular dataset as output to the user, the displayed output includingthe data values depicted as known values and the null values depicted asunknown values; receiving input from the user to populate at least aportion of the unknown values within the displayed tabular dataset withpredicted values; querying the indices for the predicted values; anddisplaying the predicted values as updated output to the user.
 22. Thenon-transitory computer readable storage medium of claim 21: whereindisplaying the tabular dataset as output to the user and displaying thepredicted values as updated output to the user comprises displaying thetabular dataset and the predicted values within a spreadsheet or tableat a Graphical User Interface (GUI); wherein the GUI further comprises aslider interface controllable by the user to specify a fill percentagefor the spreadsheet or table; and wherein receiving input from the userto populate at least a portion of the unknown values within thedisplayed tabular dataset with predicted values comprises receiving thefill percentage as input from the user via the slider interface.
 23. Thenon-transitory computer readable storage medium of claim 21, whereinquerying the indices for the predicted values comprises: generating aPredictive Query Language (PreQL) query specifying a PREDICT commandterm for each and every one of the identified plurality of null valueswithin the tabular dataset; issuing each of the generated PreQL queriesto a Predictive Query Language Application Programming Interface (PreQLAPI); and receiving a predicted result for each and every one of theidentified plurality of null values within the tabular datasetresponsive to the issued PreQL queries.
 24. A system to operate within ahost organization, the system comprising: a processor to executeinstructions stored in memory of the system; a request interface toreceive a tabular dataset from a user as input, the tabular datasethaving data values organized as columns and rows; an analysis engine toidentify a plurality of null values within the tabular dataset, the nullvalues being dispersed across multiple rows and multiple columns of thetabular dataset; the analysis engine to further generate indices fromthe tabular dataset of columns and rows, the indices representingprobabilistic relationships between the rows and the columns of thetabular dataset; the request interface to return the tabular dataset asdisplay output to the user, the display output including the data valuesdepicted as known values and the null values depicted as unknown values;the request interface to receive input from the user to populate atleast a portion of the unknown values within the displayed tabulardataset with predicted values; a query interface to query the indicesfor the predicted values; and the request interface to further returnthe predicted values as updated display output to the user.
 25. Thesystem of claim 24, further comprising: a predictive database to storethe indices; and wherein the predictive database is to execute as anon-demand cloud based service at the host organization for one or moresubscribers.
 26. The system of claim 24, further comprising: anauthenticator to verify the user as a known subscriber and to furtherverify authentication credentials presented by the known subscriber. 27.The system of claim 24, further comprising: a web-server to implementthe request interface; wherein the web-server is to receive as input, aplurality of access requests from one or more client devices from amonga plurality of customer organizations communicably interfaced with thehost organization via a network; a multi-tenant database system withpredictive database functionality to implement the predictive database;and wherein each customer organization is an entity selected from thegroup consisting of: a separate and distinct remote organization, anorganizational group within the host organization, a business partner ofthe host organization, or a customer organization that subscribes tocloud computing services provided by the host organization.